February 2026
M	T	W	T	F	S	S
	1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28

Archive for the ‘Unicode’ Category

“No mapping for the Unicode character exists in the target multi-byte code page”

Posted by jpluimers on 2021/06/24

Usually when I see this error [Wayback] “No mapping for the Unicode character exists in the target multi-byte code page” – Google Search, it is in legacy code that uses string buffers where decoding or decompressing data into.

This is almost always wrong no matter what kind of data you use, as it will depend in your string encoding.

I have seen it happen especially in these cases:

base64 decoding from string to string (solution: decode from a string stream into a binary stream, then post-process from there)
zip or zlib decompress from binary stream to string stream, then reading the string stream (solution: decompress from binary stream to binary stream, then post-process from there)

Most cases I encountered were in Delphi and C code, but surprisingly I also bumped into C# exhibiting this behaviour.

I’m not alone, just see these examples from the above Google search:

[WayBack] C# metro No mapping for the Unicode character exists in the target multi-byte code page – Stack Overflow
[WayBack] No mapping for the Unicode character exists in the target multi-byte code page – The file must be in UTF-8 or UTF-16 encoding. :: WinSCP
[WayBack] file io – WinRT No mapping for the Unicode character exists in the target multi-byte code page – Stack Overflow
[WayBack] No mapping for the Unicode character exists in the target multi-byte code page · Issue #110 · VBA-tools/VBA-Web · GitHub
Fixed issues:
- [WayBack] Error while import file · Issue #566 · HeidiSQL/HeidiSQL · GitHub
- [WayBack] EEncodingError: No mapping for the Unicode character in target multi-byte code page · Issue #515 · HeidiSQL/HeidiSQL · GitHub
[WayBack] No mapping for the Unicode character exists in the target multi-byte code page ? (Delphi XE6) – Stack Overflow (tries to decompress a string)
[WayBack] android – delphi xe6 No mapping for the Unicode character exists in the target multi-byte code page – Stack Overflow (tries to always download HTTP with the same encoding, despite it can have any encoding)
[WayBack] delphi – What could cause “No mapping for the Unicode character exists in the target multi-byte code page”? – Stack Overflow (which was a bug in the Delphi RTL method TFile.AppendAllBytes which is still present in Delphi 10.2 Tokyo, and likely newer versions)
Two artifacts of the same resolved bug in the DevExpress ExpressRichEdit Control (they do fix their bugs and release fixed versions soon):
- [WayBack] T663543 – Saving document as UTF-8 plain text doesn’t write the preamble | DevExpress Support Center
- [WayBack] T664497 – The “No mapping for the Unicode character exists in the target multi-byte code page” error occurs on loading a TXT file that doesn’t have an encoding preamble | DevExpress Support Center

–jeroen

Posted in .NET, base64, C, C#, C++, Delphi, Development, Encoding, Software Development, Unicode | Leave a Comment »

VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console

Posted by jpluimers on 2021/04/21

Every now and then it is useful to be able to do maintenance work from the ESXi console addition to the ESXi web-user interface.

I know there are many sites having this information, but many of them forgot to format the statements with code markup, so parameters with two dashes -- (each a Wayback Unicode Character ‘HYPHEN-MINUS’ (U+002D)) now have become an [Wayback] Unicode Character ‘EN DASH’ (U+2013) which is incompatible with most console programs, especially the ESXi ones (as they are Busybox based to minimise footprint).

Note you can use this small site (which runs in-browser, so does not phone home) to get the unicode code points for any string: [Wayback] What Unicode character is this ?.

Links like below (most on the vmware.com domain) have this EN DASH and make me document things on my blog instead of trying code directly from blogs or forum posts:

So below are three commands I use that have to do with the maintenance mode (the mode that for instance you can use to update an ESXi host to the latest patch level).

1. Check the maintenance mode (which returns Enabled or Disabled):
```
esxcli system maintenanceMode get
```
2. Enable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already enabled. when failed):
```
esxcli system maintenanceMode set --enable true
```
3. Disable maintenance mode (which returns nothing when succeeded, and Maintenance mode is already disabled. when failed):
```
esxcli system maintenanceMode get
```

Some examples, especially an the various output possibilities (commands in bold, output in italic):

# esxcli system maintenanceMode get
Disabled
# esxcli system maintenanceMode set --enable false
Maintenance mode is already disabled.
# esxcli system maintenanceMode set --enable true 
# esxcli system maintenanceMode get
Enabled
# esxcli system maintenanceMode set --enable true
Maintenance mode is already enabled.
# esxcli system maintenanceMode set --enable false
# esxcli system maintenanceMode get
Disabled

I made these scripts for this:

esxcli-maintenanceMode-show.sh:

#!/bin/sh
esxcli system maintenanceMode get

esxcli-maintenanceMode-enter.sh:

#!/bin/sh
esxcli system maintenanceMode set --enable true

esxcli-maintenanceMode-exit.sh:

#!/bin/sh
esxcli system maintenanceMode set --enable false

Note I have not checked the exit codes for these esxcli commands yet, but did blog about how to do that: Busybox sh (actually ash derivative dash): checking exit codes.

–jeroen

Posted in BusyBox, Development, Encoding, ESXi6, ESXi6.5, ESXi6.7, ESXi7, Power User, Software Development, Unicode, Virtualization, VMware, VMware ESXi | Leave a Comment »

(mostly ASCII) List of emoticons – Wikipedia

Posted by jpluimers on 2021/03/17

Most searches for “ASCII emoticons” get you Unicode ones:

Luckily most are ASCII in List of emoticons – Wikipedia.

There are also shortcodes, which do not visually represent an emoji, but usually get translated to the image or Unicode character.

A few lists on them:

–jeroen

Posted in ASCII, Development, Encoding, LifeHacker, Power User, Software Development, Unicode | Leave a Comment »

Unicode is hard, also for the Delphi compiler and IDE

Posted by jpluimers on 2020/10/13

The Delphi compiler does not see a unicode non-breaking space (0x00A0 as whitespace, and the Delphi IDE does not warn you about it: [WayBack] Delphi revelations #2 – Space characters are not just space characters.

Given that this character was introduced in 1993, I wonder how the compiler tests look like.

These also will not be recognised as whitespace:

Related, as many other tools also do not properly support various whitespace characters:

Via: [WayBack] A Delphi “Aha” experience – Kim Madsen – Google+

–jeroen

Posted in Delphi, Development, Software Development, Unicode | Leave a Comment »

Hamburger menu character on unicode: use U+2261 instead of U+2630

Posted by jpluimers on 2020/01/27

Not all fonts have Unicode character ☰ [WayBack] Unicode Character ‘TRIGRAM FOR HEAVEN’ (U+2630) as it is in a less common block.

More fonts have Unicode character ≡ [WayBack] Unicode Character ‘IDENTICAL TO’ (U+2261)

The latter is slightly shorter and slightly narrower than the former, but works in way more places.

Via [WayBack] html – Unicode ☰ hamburger not displaying in Android & Chrome – Stack Overflow

I’ve worked around this problem by using the UNICODE character UNICODE U+2261 (8801), ≡ IDENTICAL TO as illustrated below rather than the UNICODE U+2630 (9776) ☰ TRIGRAM FOR HEAVEN which

–jeroen

Posted in Development, Encoding, LifeHacker, Power User, Software Development, Unicode | Leave a Comment »

Delphi, decoding files to strings and finding line endings: some links, some history on Windows NT and UTF/UCS encodings

Posted by jpluimers on 2019/12/31

A while back there were a few G+ threads sprouted by David Heffernan on decoding big files into line-ending splitted strings:

[WayBack] Having been a little underwhelmed by the performance of TStreamReader when reading huge text files line by line, I attempted to roll my own. I managed … – David Heffernan – Google+ where he compares the speed with Python (which runs circles around equivalent Delphi code)
[WayBack] I just read this in TEncoding.GetBufferEncoding: function ContainsPreamble(const Buffer, Signature: array of Byte): Boolean; var I: Integer; … – David Heffernan – Google+ (worse: Delphi 209 and up have 3 different implementations of the ContainsPreamble function)

Code comparison:

Python:

with open(filename, 'r', encoding='utf-16-le') as f:
  for line in f:
    pass

Delphi:

for Line in TLineReader.FromFile(filename, TEncoding.Unicode) do
  ;

This spurred some nice observations and unfounded statements on which encodings should be used, so I posted a bit of history that is included below.

Some tips and observations from the links:

Good old text files are not “good” with Unicode support, neither are TextFile Device Drivers; nobody has written a driver supporting a wide range of encodings as of yet.
Good old text files are slow as well, even with a changed SetTextBuffer
When using the TStreamReader, the decoding takes much more time than the actual reading, which means that [WayBack] Faster FileStream with TBufferedFileStream • DelphiABall does not help much
TStringList.LoadFromFile, though fast, is a memory allocation dork and has limits on string size
Delphi RTL code is not what it used to be: pre-Delphi Unicode RTL code is of far better quality than Delphi 2009 and up RTL code
Supporting various encodings is important
EBCDIC days: three kinds of spaces, two kinds of hyphens, multiple codepages
Strings are just that: strings. It’s about the encoding from/to the file that needs to be optimal.
When processing large files, caching only makes sense when the file fits in memory. Otherwise caching just adds overhead.
On Windows, if you read a big text file into memory, open the file in “sequential read” mode, to disable caching. Use the FILE_FLAG_SEQUENTIAL_SCAN flag under Windows, as stated at [WayBack] How do FILE_FLAG_SEQUENTIAL_SCAN and FILE_FLAG_RANDOM_ACCESS affect how the operating system treats my file? – The Old New Thing
Python string reading depends on the way you read files (ASCII or Unicode); see [WayBack] unicode – Python codecs line ending – Stack Overflow

Though TLineReader is not part of the RTL, I think it is from [WayBack] For-in Enumeration – ADUG.

Encodings in use

It doesn’t help that on the Windows Console, various encodings are used:

Most tools still use Code Page 437 (from the good old DOS days, also known as CP437, OEM-US, OEM 437, PC-8, or DOS Latin US.)
Unicode aware applications often use UTF-8
A minority (but growing because of PowerShell: [WayBack] utf 8 – Changing PowerShell’s default output encoding to UTF-8 – Stack Overflow) of tools uses UTF-16 because Windows Unicode support started with UCS-2 with an in-memory little-endian representation (.REG files, MSXML, SQL Server Management Studio)

Good reading here is [WayBack] c++ – What unicode encoding (UTF-8, UTF-16, other) does Windows use for its Unicode data types? – Stack Overflow

Encoding history

+A. Bouchez I’m with +David Heffernan here:

At its release in 1993, Windows NT was very early in supporting Unicode. Development of Windows NT started in 1990 where they opted for UCS-2 having 2 bytes per character and had a non-required annex on UTF-1.

UTF-1 – that later evolved into UTF-8 – did not even exist at that time. Even UCS-2 was still young: it got designed in 1989. UTF-8 was outlined late 1992 and became a standard in 1993

Some references:

[WayBack] Windows NT and VMS: The Rest of the Story | IT Pro

[WayBack] Windows NT – Wikipedia

[WayBack] History: UTF-8 – Wikipedia

[WayBack] UCS History: Universal Coded Character Set – Wikipedia

[WayBack] UTF-1 – Wikipedia

–jeroen

Read the rest of this entry »

Posted in Delphi, Development, Encoding, PowerShell, PowerShell, Python, Scripting, Software Development, The Old New Thing, Unicode, UTF-16, UTF-8, Windows Development | Leave a Comment »

Unicode ligatures: not all software does normalised search forgetting ffi

Posted by jpluimers on 2019/06/26

Via a private share, I found out that some software forgets to perform a Unicode normalisation when doing a search.

That means that ligatures do not match the non-ligatures in for instance these words:

“ff” and “ﬀ”, as in “difference” versus “diﬀerence”
“fi” and “ﬁ” as in “notification” versus “notiﬁcation”.

For more information, read [WayBack] Unicode equivalence – Wikipedia and make sure you know about these normal forms:

NFD
Normalization Form Canonical Decomposition Characters are decomposed by canonical equivalence, and multiple combining characters are arranged in a specific order.

NFC
Normalization Form Canonical Composition Characters are decomposed and then recomposed by canonical equivalence.

NFKD
Normalization Form Compatibility Decomposition Characters are decomposed by compatibility, and multiple combining characters are arranged in a specific order.

NFKC
Normalization Form Compatibility Composition Characters are decomposed by compatibility, then recomposed by canonical equivalence.

NFD Normalization Form Canonical Decomposition	Characters are decomposed by canonical equivalence, and multiple combining characters are arranged in a specific order.
NFC Normalization Form Canonical Composition	Characters are decomposed and then recomposed by canonical equivalence.
NFKD Normalization Form Compatibility Decomposition	Characters are decomposed by compatibility, and multiple combining characters are arranged in a specific order.
NFKC Normalization Form Compatibility Composition	Characters are decomposed by compatibility, then recomposed by canonical equivalence.

–jeroen

Posted in Development, Encoding, Software Development, Unicode | Leave a Comment »

I’ve given up on entering non-ASCII characters when entering data on-line

Posted by jpluimers on 2019/06/17

I live in a street that has a non-ASCII character in it: Pyreneeën.

I’ve reverted back to entering the street name as plain ASCII for a simple reason:

Too often the ë gets mangled into encoding gibberish, similar to the é example in [WayBack] When Good Characters Go Bad: A Guide to Diagnosing Character Display Problems as these characters are very near both in UTF-8 and in the [WayBack] Unicode Characters in the Latin-1 Supplement Block:

UTF-8 0xC3 0xA9: [WayBack] Unicode Character ‘LATIN SMALL LETTER E WITH ACUTE’ (U+00E9)
UTF-8 0xC3 0xAB: [WayBack] Unicode Character ‘LATIN SMALL LETTER E WITH DIAERESIS’ (U+00EB)

I’ve seen these encodings, where only the top encoding is correct; the degeneration gets worse moving downwards, a classic Mojibake:

# encoded UTF-8 (hex.)

0 ë 0xC3 0xAB

1 Ã« 0xC3 0x83 0xC2 0xAB

2 ÃÂ« 0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0xAB

3 ÃÂÃÂ« 0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0xAB

4 ÃÂÃÂÃÂÃÂ« 0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x82 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0xAB

5 ë 0x26 0x65 0x75 0x6d 0x6c 0x3b

#	encoded	UTF-8 (hex.)
0	ë	`0xC3 0xAB`
1	Ã«	`0xC3 0x83 0xC2 0xAB`
2	ÃÂ«	`0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0xAB`
3	ÃÂÃÂ«	`0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0xAB`
4	ÃÂÃÂÃÂÃÂ«	`0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0x83 0xC3 0x83 0xC2 0x83 0xC3 0x82 0xC2 0x82 0xC3 0x83 0xC2 0x82 0xC3 0x82 0xC2 0xAB`
5	ë	`0x26 0x65 0x75 0x6d 0x6c 0x3b`

The last one seldomly happens, the first one relatively often, just like [Archive.is] fd.nl did a while on their finanancial pages.

These mistakes become sort of understandable (but not forgivable) when you look at the below table-fragment (the full table is at[WayBack] Unicode/UTF-8-character table – starting from code position 0080).

Read the rest of this entry »

Posted in Development, Encoding, Mojibake, Power User, Software Development, Unicode, Web Browsers | Leave a Comment »

Getting rid of trailing line-endings in the draw.io web interface

Posted by jpluimers on 2018/12/03

One of the things that bugged me for a long time is that every now and then for some shapes, when editing their text, the draw.io web interface puts in trailing line feeds after the text, messing up layout.

The easiest way to work around it is by searching inside the diagram XML for ", then replacing that with a ".

(the above code got screwed by WordPress.com saving it, so the search is in this small gist below)

This behaviour is intermittent on the drawio MacOS desktop app.

view raw

draw.io-line-ending.txt

hosted with ❤ by GitHub

–jeroen

Every now and then the visual editor at https://t.co/HhHS5rzG2X screws up when saving. For now @WordPress needs to check why the below inside <code> tags gets translated to a line feed, despite the text view having it correctly escaped:https://t.co/pv30Lg9NZW


"

— Jeroen Wiert Pluimers @wiert (@jpluimers) July 25, 2018

The behaviour is intermittent in the Mac desktop app.

— Jeroen Wiert Pluimers @wiert (@jpluimers) December 3, 2018

a non intermittend behaviour there, is pasting multi-line text.

When you paste it inside a shape, all line-endings are removed every time you try.

The workaround is to double click on the text inside a shape, then paste: now the line endings are preserved.

— Jeroen Wiert Pluimers @wiert (@jpluimers) December 3, 2018

Posted in Cloud Apps, Development, draw.io, Encoding, Internet, Power User, Software Development, Unicode | Leave a Comment »

Unicode spaces

Posted by jpluimers on 2018/09/25

For my link archive:

Via: [WyBack] Are there blank characters in unicode that have the same widths as period, comma and digits? – Lars Fosdal – Google+

Answer: no, though better fonts have period, comma, colon, semicolon and other punctuations the same width as the punctuation space.

The use-case:

Lars Fosdal

I wanted right justified text without having to do custom positioning/drawing – where the decimal zero is white space.

F.x. here 12 instead of 12.0

9.5
11.6
12 <– #$2008 and #$2007
13.4

I.e. PunctuationSpace and FigureSpace

I don’t want to deal with positioning/rendering since it happens inside a third party component.

–jeroen

Posted in Development, Encoding, Font, Power User, Software Development, Unicode | Leave a Comment »

« Previous Entries

Next Entries »

	Attila Kovacs on Crowbarring Windows 95 into Wi…
	Jeroen Wiert Pluimer… on Does Odido (the old T-Mobile N…
	Lars Fosdal on Security alarm provider Woonve…
	Thomas Mueller on Question got closed in May 202…
	Thaddy de Koning on Formulier voor bewindvoerders…

The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

Twitter Updates

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘Unicode’ Category

“No mapping for the Unicode character exists in the target multi-byte code page”

VMware ESXi 6 and 7: checking and setting/clearing maintenance mode from the console

(mostly ASCII) List of emoticons – Wikipedia

Unicode is hard, also for the Delphi compiler and IDE

Hamburger menu character on unicode: use U+2261 instead of U+2630

Delphi, decoding files to strings and finding line endings: some links, some history on Windows NT and UTF/UCS encodings

Encodings in use

Encoding history

Unicode ligatures: not all software does normalised search forgetting ffi

I’ve given up on entering non-ASCII characters when entering data on-line

Getting rid of trailing line-endings in the draw.io web interface

Unicode spaces

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

Subscribe

Archives

Recent Comments

Recent Posts

Blog Stats

Meta title

Tag Cloud Title

Top Clicks

Top Posts

My badges

My Flickr Stream

Pages

All categories

Email Subscription

Archive for the ‘Unicode’ Category

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Encodings in use

Encoding history

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this:

Rate this:

Share this: