Get the new [Wayback/Archive] Delphi Thread Safety Patterns eBook at a discount while it is hot:
Use Coupon Code: DTSPATT10 at checkout to get a $10 discount.
This promotional offer is valid through June 14.
Archive for the ‘Windows-1252’ Category
Get it at a discount while it is hot: Delphi Thread Safety Patterns eBook by Dalija Prasnikar and Neven Prasnikar Jr.
Posted by jpluimers on 2022/06/01
Posted in Delphi, Development, Encoding, ISO-8859, ISO8859, Mojibake, Multi-Threading / Concurrency, Software Development, UTF-8, Windows-1252 | Leave a Comment »
In this day and age, web sites with delivery back-ends still have Unicode issues: at least @Woonveilig, @Medireva and @PostNL still have trouble
Posted by jpluimers on 2022/02/09
Nowadays, some 35 years after the first Unicode ideas got drafted and 30+ years after the Unicode Consortium saw the light, UTF-8 is served my more than 95% of the web as shown in yesterday’s post UTF-8 web adoption is huge, closing 100%, but only soured up since around 2006..
I mentioned this:
It means that nowadays there is a very small chance you will see mangled characters (what Japanese call mojibake) when you’re surfing the web.
Serving UTF8 does not mean no unicode problems.
Below are some issues that happened not too long ago and still happen. I have reported them to all parties involved through web-care, but no response whatsoever, and this is bad: Unicode support beyond basic ASCII for the below systems are still broken even for relatively simple non-ASCII characters based in diacritics decorating a standard ASCII character.
Yes, I know the realm of encoding and code pages is a mess, especially when handling data in multiple layers of an application stack. That’s why I wrote this post in the first place, and have a whole encoding category of blog posts plus a Mojibake subset.
Share this:
Posted in Communications Development, CP850, Dark Pattern, Development, Encoding, ISO-8859, ISO8859, Mojibake, Software Development, Unicode, User Experience (ux), UTF-16, UTF-8, Windows-1252 | Leave a Comment »
Which encoding failure did encode “vóór” into “v3/43/4r”? – Stack Overflow
Posted by jpluimers on 2020/02/24
From quite some time ago, but still very relevant as encoding issues keep occurring:
A while ago, I saw the text “v3/43/4r” in a document.I know it comes from “vóór” (the acute accent emphasises in Dutch), and wonder which encoding failure was applied to get this wrong.
Source: [WayBack] Which encoding failure did encode “vóór” into “v3/43/4r”? – Stack Overflow
From the [WayBack] answer by rodrigo:
- ó: is U+00F3, and occupies the same codepoint (0xF3) in a lot of different encodings (most ISO-8859-* and most western Windows-*).
- In CP850 the codepint 0xF3 is ¾ (U+00BE), that is the three-quarters character. It is the same in other, less used, codepages (CP775, CP856, CP857, CP858).
- The ¾ is sometimes transliterated to 3/4 when the character is not directly available.
And there you are! “vóór” -> “v¾¾r” -> “v3/43/4r”.
The first part (ó -> ¾) is the usual corruption of ANSI vs. OEM codepages in the Western Windows versions (in my country ANSI=Windows-1252, OEM=CP850). You can see it easily creating a file with NOTEPAD, writing
vóórand dumping it in a command prompt withtype.
–jeroen
Share this:
Posted in CP850, Development, Encoding, Software Development, UTF-8, UTF8, Windows-1252 | Leave a Comment »
Long read about Unicode: You, Me And The Emoji: Character Sets, Encoding And Emoji – Smashing Magazine
Posted by jpluimers on 2017/11/07
A well worth long rad:
We all recognize emoji. They’ve become the global pop stars of digital communication. But what are they, technically speaking? And what might we learn by taking a closer look at these images, characters, pictographs… whatever they are 🤔 (Thinking Face). We will dig deep to learn about how these thingamajigs work. Please note: Depending on your browser, you may not be able to see all emoji featured in this article (especially the Tifinagh characters). Also, different platforms vary in how they display emoji as well. That’s why the article always provides textual alternatives. Don’t let it discourage you from reading though! Now, let’s start with a seemingly simple question. What are emoji?
[WayBack] You, Me And The Emoji: Character Sets, Encoding And Emoji – Smashing Magazine
Via: [WayBack] Everything you ever wanted to know about characters, encodings, glyphs… and, oh yeah, emoji: bit.ly/2fNKeW3Long, rewarding read. – Ilya Grigorik – Google+
Here is just the ToC:
TABLE OF CONTENTS LINK
–jeroen
Share this:
Posted in ASCII, Development, Encoding, ISO-8859, ISO8859, Shift JIS, Unicode, UTF-16, UTF-8, UTF16, UTF8, Windows-1252 | Leave a Comment »
Looking for more examples of Unicode/Ansi oddities in Delphi 2009+
Posted by jpluimers on 2017/09/25
At the end of April 2014, Roman Yankovsky started a nice [Wayback] discussion on Google+ trying to get upvotes for [Wayback] QualityCentral Report #: 124402: Compiler bug when comparing chars.
His report basically comes down to that when using Ansi character literals like #255, the compiler treats them as single-byte encoded characters in the current code page of your Windows context, translates them to Unicode, then processes them.
The QC report has been dismissed as “Test Case Error” (within 15 minutes of stating “need more info”) by one of the compiler engineers, directing to the [Wayback] UsingCharacterLiterals section of Delphi in a Unicode World Part III: Unicodifying Your Code where – heaven forbid – they suggest to replace #128 with the Euro-Sign literal.
I disagree, as the issue happens without any hint or warning whatsoever, and causes code that compiles fine in Delphi <= 2007 to fail in subtle ways on Delphi >= 2009.
The compiler should issue a hint or warning when you potentially can screw up. It doesn’t. Not here.
Quite a few knowledgeable Delphi people got involved in the discussion:
Share this:
Posted in Ansi, ASCII, Conference Topics, Conferences, CP437/OEM 437/PC-8, Delphi, Delphi 2006, Delphi 2007, Delphi 2009, Delphi 2010, Delphi 7, Delphi XE, Delphi XE2, Delphi XE3, Delphi XE4, Delphi XE5, Delphi XE6, Development, Encoding, Event, ISO-8859, Missed Schedule, QC, SocialMedia, Software Development, Unicode, UTF-8, Windows-1252, WordPress | Leave a Comment »
How can I get the default code page for a locale? – The Old New Thing
Posted by jpluimers on 2017/06/20
Ask GetLocaleInfo (example function GetAnsiCodePageForLocale included): [WayBack] How can I get the default code page for a locale? – The Old New Thing
UINT GetAnsiCodePageForLocale(LCID lcid) { UINT acp; int sizeInChars = sizeof(acp) / sizeof(TCHAR); if (GetLocaleInfo(lcid, LOCALE_IDEFAULTANSICODEPAGE | LOCALE_RETURN_NUMBER, reinterpret_cast<LPTSTR>(&acp), sizeInChars) != sizeInChars) { // Oops - something went wrong } return acp; }And even though you didn’t ask, you can use
LOCALE_IDEFAULTCODEPAGEto get the OEM code page for a locale.Bonus gotcha: There are a number of locales that are Unicode-only. If you ask the
GetLocaleInfofunction and ask for their ANSI and OEM code pages, the answer is “Um, I don’t have one.” (You get zero back.)
Related:
- [WayBack] windows – Why ANSI Code-Page and Console Code-Page are different? – Stack Overflow
- [WayBack] LOCALE_IDEFAULT* Constants – Windows applications | Microsoft Docs
- [WayBack] GetLocaleInfoA function | Microsoft Docs
- [WayBack] GetLocaleInfoW function | Microsoft Docs
- [WayBack] GetLocaleInfoEx function | Microsoft Docs
–jeroen
Share this:
Posted in Development, Encoding, internatiolanization (i18n) and localization (l10), Software Development, The Old New Thing, Windows Development, Windows-1252 | 2 Comments »
Encoding is hard… so how did the single quote become a circumflexed a followed by Euro sign and trade mark?
Posted by jpluimers on 2016/10/04
A while ago (in fact more than a year), I posted Encoding is hard… go G+ with the below picture.
[Wayback] ftfy (“fixes text for you”, a parody on “fixed that for you”) [Wayback] fixes it, but:
How did the single quote become “’“?
Actually, because of a a common “beautification” of many Office suites (Microsoft and Open alike), the single quote was a special one: a Unicode Character ‘RIGHT SINGLE QUOTATION MARK’ (U+2019) which in UTF-8 is encoded as 0xE2 0x80 0x99.
Share this:
Posted in Development, Encoding, ftfy, ISO-8859, ISO8859, Mojibake, Software Development, Unicode, UTF-8, UTF8, Windows-1252 | Leave a Comment »
Some interesting encoding/Unicode/text articles on kunststube and links for test files of various encodings
Posted by jpluimers on 2016/08/17
After yesterdays post on Testing and static methods don’t go well together, I read around on Source (kunststube [WayBack]) a bit more and found these very nice articles on encoding,Unicode and text:
- What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text [WayBack]
- The Great Escapism (or: What you Need to Know to Work With Text Within Text) [WayBack]
- Handling Unicode Front to Back in a Web App [WayBack]
Related on those, some other nice readings:
- Is there a set of “Lorem ipsums” files for testing character encoding issues? – Stack Overflow [WayBack]
- International Components for Unicode: ICU User Guide [WayBack]
- International Components for Unicode ː Repository Browser: repos: icu/data/trunk/charset/data/ucm
- ftp://ftp.unicode.org/Public/MAPPINGS [WayBack]
-
Notes on contents of the MAPPING directory: EASTASIA: This directory is obsolete. ETSI: ETSI GSM 03.38 7-bit default alphabet mapping. ISO8859: These are the mapping tables of the ISO 8859 series (1 - 16). OBSOLETE: Obsolete and unsupported mapping tables for historical and archival purposes only. VENDORS: Miscellaneous mapping tables for small codesets, typically provided by vendors. The majority of current, useful tables are here.
-
–jeroen
Share this:
Posted in Ansi, ASCII, CP437/OEM 437/PC-8, Development, EBCDIC, Encoding, ISO-8859, ISO8859, Shift JIS, Software Development, Unicode, UTF-16, UTF-8, UTF16, UTF8, Windows-1252 | Leave a Comment »
Windows key character that displays on non-Windows systems (like Mac)
Posted by jpluimers on 2014/08/08
Though there is a Unicode character for the Apple Command Key, there is none for the Windows Key.
The Windows font WinDings does have a character 255 for it, but that font usually is not installed on non-Windows systems. There it will look like Unicode Character ‘LATIN SMALL LETTER Y WITH DIAERESIS’ (U+00FF)
This Unicode code point comes closest to the Windows key: Unicode Character ‘SQUARED PLUS’ (U+229E) and is used by Windows Key page on WikiPedia.
- The WinDings character looks like this: ÿ
(non no Windows systems, it will look like an y with two dots on it: ÿ) - The Unicode Codepoint U+229E like this: ⊞
Not a complete match, but pretty close.
The Unicode code points for Mac modifier keys are these:
- Enter: Unicode Character ‘LEFTWARDS ARROW WITH HOOK’ (U+21A9)
which looks like ↩ - Shift: Unicode Character ‘UPWARDS WHITE ARROW’ (U+21E7)
which looks like ⇧ - Command: Unicode Character ‘PLACE OF INTEREST SIGN’ (U+2318) which looks like ⌘
- Option/Alt: Unicode Character ‘OPTION KEY’ (U+2325)
which looks like ⌥ - Esc: Unicode Character ‘BROKEN CIRCLE WITH NORTHWEST ARROW’ (U+238B)
which looks like ⎋ - Eject: Unicode Character ‘EJECT SYMBOL’ (U+23CF)
which looks like ⏏
–jeroen
Share this:
Posted in Development, Encoding, Mac, Mac OS X / OS X / MacOS, Mac OS X 10.4 Tiger, Mac OS X 10.5 Leopard, Mac OS X 10.6 Snow Leopard, Mac OS X 10.7 Lion, MacBook Retina, MacBook-Air, MacBook-Pro, OS X 10.8 Mountain Lion, Power User, Software Development, Unicode, Windows, Windows 7, Windows 8, Windows Server 2003, Windows Server 2003 R2, Windows Vista, Windows XP, Windows-1252 | Leave a Comment »
Recommended reads when dealing with Character Encodings in software
Posted by jpluimers on 2014/05/06
Apart from the mandatory Joel on Software article about Unicode and Character sets, these two articles are of great value too:
Fun to read from that blog is the Historical Technology section including this article:
–jeroen
PS: The mandatory one is The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) – Joel on Software.
Share this:
Posted in .NET, Ansi, ASCII, CP437/OEM 437/PC-8, Delphi, Development, EBCDIC, Encoding, ISO-8859, ISO8859, Shift JIS, Software Development, Unicode, UTF-8, UTF8, Windows-1252 | Leave a Comment »
The Wiert Corner - irregular stream of stuff 