Bad surprise of the day: SysUtils.TEncoding in XE2+ defaults to ANSI, while in XE it defaulted to UTF-8 .Among other things this means that TStringList… – Eric Grange – Google+
+Stefan Glienke Indeed, you’re right. The issue must be deeper somewhere. Don’t have time to investigate too much, I’m bypassing the RTL now (also have to work around the limitation that for utf-8 the TEncoding.GetString method returns an empty string if one character in the buffer isn’t utf-8)
I totally missed the passing of Michael Scott Kaplan some 2 years ago, so a belated R.I.P. is in place.
Obituary for Michael Kaplan, Michael Scott Kaplan, 45, passed away Wednesday, October 21, 2015, in Redmond, WA, after a brave battle with MS for 25 years. He was a lead software developer for Microsoft.
Michael was the leading source on i18n, L10N, Unicode, sorting, normalisation and other things having to do with languages, representations and writing.
Besides that he was a really nice guy of which I enjoyed his MSDN materials.
His report basically comes down to that when using Ansi character literals like #255, the compiler treats them as single-byte encoded characters in the current code page of your Windows context, translates them to Unicode, then processes them.
I disagree, as the issue happens without any hint or warning whatsoever, and causes code that compiles fine in Delphi <= 2007 to fail in subtle ways on Delphi >= 2009.
The compiler should issue a hint or warning when you potentially can screw up. It doesn’t. Not here.
Quite a few knowledgeable Delphi people got involved in the discussion:
Notes on contents of the MAPPING directory:
EASTASIA:
This directory is obsolete.
ETSI:
ETSI GSM 03.38 7-bit default alphabet mapping.
ISO8859:
These are the mapping tables of the ISO 8859 series (1 - 16).
OBSOLETE:
Obsolete and unsupported mapping tables for historical
and archival purposes only.
VENDORS:
Miscellaneous mapping tables for small codesets, typically provided
by vendors. The majority of current, useful tables are here.
One of the features that bites me over and over again is the ZEROBASEDSTRINGS that got introduced in Delphi XE3 and is by default ON in mobile compilers and OFF in Desktop compilers.
Back then, Mark Edington showed a small example of the effects:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
The XE3 RTL source code has been refactored to be string index base agnostic. In most cases this is done by utilizing string helper functions which are always zero based.
When it is necessary to traverse a string, the Char[] property is often used to access the individual characters without concern for the current state of the compiler with respect to zero based strings.
In addition, the “Low” and “High” standard functions can now be passed a string variable to provide further flexibility as needed.
When zero based strings are enabled, Low(string) will return 0, otherwise it will return 1. Likewise, High() returns a bounds adjusted length variation.
The problem is the non-existent forward compatibility of the other compilers (Delphi XE2 and lower).
So if you have library code that needs to work in Delphi versions, you cannot use the High and Low to make the code ZEROBASEDSTRINGS neutral.
Many Delphi developers regularly skip many Delphi versions, so these are still popular:
Delphi XE1 and XE2 (the last 2 compilers before Delphi really started to support mobile)
Delphi 2007 (the last non-Unicode Delphi compiler)
Delphi 7 (the last non-Galileo IDE)
The result is that library code is full of conditionan IF/IFDEF blocks like these:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
when Char or String data is part of the equation, set the CharSet property to CharSet.Auto. This causes the CLR to use the appropriate character set based on the host OS. If you don’t explicitly set the CharSet property, then its default is CharSet.Ansi. This default is unfortunate because it negatively affects the performance of text parameter marshaling for interop calls made on Windows 2000, Windows XP, and Windows NT®.
The only time you should explicitly select a CharSet value of CharSet.Ansi or CharSet.Unicode, rather than going with CharSet.Auto, is when you are explicitly naming an exported function that is specific to one or the other of the two flavors of Win32 OS. An example of this is the ReadDirectoryChangesW API function, which exists only in Windows NT-based operating systems and supports Unicode only; in this case you should use CharSet.Unicode explicitly.
The Wiert Corner - irregular stream of stuff 