When someone writes UTF-8 and UTF-16 strings to the same file in binary format without converting between them…
Posted by jpluimers on 2017/06/21
A while ago, I had to fix some stuff in an application that would write – using a binary mechanism – UTF-8 and UTF-16 strings (part of it XML in various flavours) to the same byte stream without converting between the two encodings.
Some links that helped me investigate what was wrong, choose what encoding to use for storage and fix it:
- unicode – How to Convert Ansi to UTF 8 with TXMLDocument in Delphi – Stack Overflow
- delphi – What should I use? UTF8 or UTF16? – Stack Overflow
- IXMLDocument.SaveToStream does not always use UTF-16 encoding | Marc Durdin’s Blog
- delphi – Length() vs Sizeof() on Unicode strings – Stack Overflow
TEncoding.UTF8.GetBytes: utf 8 – String to byte array in UTF-8? – Stack OverflowSysUtils.ByteLength()is not the nicest function, so use only for a quick-fix Delphi Unicode String Length in Bytes – Stack OverflowAnsiString,UnicodeStringet al: String Types (Delphi) – RAD Studio- Question about type identity – delphi
- On the type compatibility in Delphi | The Programming Works
- Joe White’s Blog » Blog Archive » Grammar details of Delphi’s “type type” feature
- Type Compatibility and Identity (Delphi) – RAD Studio
- Delphi “type types”: similar types but not the same type identity, some examples.
- Converting in various Delphi versions (pre-Unicode and post-Unicode versions): Delphi: Encoding Strings as Python do – Stack Overflow
- UTF-8 Conversion Routines – RAD Studio
–jeroen
The Wiert Corner - irregular stream of stuff 
Arnaud Bouchez said
UTF-16LE or UTF-16BE? ;) I hope you choose to store every text with UTF-8 encoding in your binary stream.
KMorwath said
Or at least you have a BOM or other metadata to tell readers what to expect… but I see very few cases when mixing encodings makes sense.
jpluimers said
Because other tooling involved, I cannot add a BOM, but it’s at least UTF-8 output now.