UTF-8 support for single byte character sets is beta in Windows and likely breaks a lot of applications not expecting this (via Unicode in Microsoft Windows: UTF-8 – Wikipedia)
Posted by jpluimers on 2018/12/04
Uh-oh: [WayBack] Unicode in Microsoft Windows: UTF-8 – Wikipedia:
Microsoft Windows has a code page designated for UTF-8, code page 65001. Prior to Windows 10 insider build 17035 (November 2017),[7] it was impossible to set the locale code page to 65001, leaving this code page only available for:
- Explicit conversion functions such as MultiByteToWideChar
- The Win32 console command
chcp 65001to translate stdin/out between UTF-8 and UTF-16.This means that “narrow” functions, in particular
fopen, cannot be called with UTF-8 strings, and in fact there is no way to open all possible files usingfopenno matter what the locale is set to and/or what bytes are put in the string, as none of the available locales can produce all possible UTF-16 characters.On all modern non-Windows platforms, the string passed to
fopenis effectively UTF-8. This produces an incompatibility between other platforms and Windows. The normal work-around is to add Windows-specific code to convert UTF-8 to UTF-16 using MultiByteToWideChar and call the “wide” function.[8] Conversion is also needed even for Windows-specific api such asSetWindowTextsince many applications inherently have to use UTF-8 due to its use in file formats, internet protocols, and its ability to interoperate with raw arrays of bytes.There were proposals to add new API to portable libraries such as Boost to do the necessary conversion, by adding new functions for opening and renaming files. These functions would pass filenames through unchanged on Unix, but translate them to UTF-16 on Windows.[9] This would allow code to be “portable”, but required just as many code changes as calling the wide functions.
With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a “Beta: Use Unicode UTF-8 for worldwide language support” checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling “narrow” functions, including
fopenandSetWindowTextA, with UTF-8 strings. Microsoft claims this option might break some functions (a possible example is _mbsrev[10]) as they were written to assume multibyte encodings used no more than 2 bytes per character, thus until now code pages with more bytes such as GB 18030 (cp54936) and UTF-8 could not be set as the locale.[11]
- Jump up^ [WayBack] “UTF-8 in Windows”. Stack Overflow. Retrieved July 1, 2011.
- Jump up^ [WayBack] “Boost.Nowide”.
- Jump up^ [WayBack] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/strrev-wcsrev-mbsrev-mbsrev-l
- Jump up^ [WayBack] “Code Page Identifiers (Windows)”. msdn.microsoft.com.
Via [WayBack] Microsoft Windows Beta UTF-8 support for Ansi API could break things. Wiki Article of the Change… – Tommi Prami – Google+
Related, as handling encoding is hard, especially if it is changed or not your default:
- “windows” “17134” “utf8” – Google Search and “windows” “17134” “65001” “error” – Google Search breaks lots of applications
- [RSP-21814] Microsoft Windows Beta UTF-8 support for Ansi API could break things – Embarcadero Technologies
[WayBack]
One of out customer had selected that and we started to experience very weird problems and took some time to find out why it misbehaves.
None of the application could connect to Firebird SQL server (Ours or third party) successfully.
So would be smart to go through all tooling and code with that setting, we never know what M$oft will do with that, will it ever be released or will it soon be default for all.
- [WayBack] windows – Error starting SQL Server 2017 service. Error Code 3417 – Database Administrators Stack Exchange
- [WayBack] windows – How do you get getLine to accept unicode characters? – Stack Overflow
- [WayBack] [FIRE-23012] Windows 10’s UTF8 beta breaks URL parsing – Firestorm Bug Tracker
- [WayBack] Py_Initialize: can’t initialize sys standard streams, unknown encoding: 65001 · Issue #7674 · conda/conda · GitHub
- [WayBack] Blinking window while press TAB on Windows 10 · Issue #7863 · PowerShell/PowerShell · GitHub
- [WayBack] cl.exe breaks with case sensitive paths – Developer Community
–jeroen
The Wiert Corner - irregular stream of stuff 
rvelthuis said
“likely breaks a lot of applications not expecting this”. I am not so sure. Windows apps before the dates mentioned will already do the conversion, if necessary, or simply pass a non-UTF-8 string, so I don’t think it will break a lot of existing code. But as you quote: “Microsoft claims this option might break some functions” (emphasis mine). Not lots, just some. And only those who do not use the wide functions already.
jpluimers said
Breaking some functions, does not imply that only some applications are affected. Given how hard handling localisation related things are, even a few functions breaking can cause a lot of harm.
Just searching for “windows” “17134” “65001” “error”, already returns thousands of hits.