The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for the ‘Encoding’ Category

ASCII is not just an RFC. It is an Internet Standard, but only recently.

Posted by jpluimers on 2015/01/29

When people tell you that ASCII is not an Internet Standard but an RFC. They are wrong. They used to be right though. Until 2015-01-12, when IETF declared the RFC 20 to be an Internet Standard: status-change-rfc20-ascii-format-to-standard-00.

So after more than 45 years (like many good things, the ASCII RFC is from 1969), it is not just an American Standard but an Internet Standard (:

Thanks Lauren Weinstein for sharing and Kristian Köhntopp for pointing to the reclassification.

–jeroen

via: ASCII – Wikipedia, the free encyclopedia.

Posted in ASCII, Development, Encoding, History, Software Development | Leave a Comment »

Great Unicode presentation by

Posted by jpluimers on 2015/01/21

Stefan Heymann did a great presentation Character Sets and Unicode in Firebird at fbcon11. About 90% of it is not about Firebird, but about Unicode: a highly recommended presentation.

There is also a PDF version of the same presentation for easier reading/searching.

If you like Firebird, there is a whole bunch of Firebird related presentations from various authors shared by MindTheBird.

–jeroen

Posted in Ansi, Database Development, Development, Encoding, Firebird, ISO-8859, ISO8859, Software Development, Unicode, UTF-8, UTF8 | Leave a Comment »

Delphi: ZEROBASEDSTRINGS and maintaining cross-version Delphi libraries

Posted by jpluimers on 2015/01/14

One of the features that bites me over and over again is the ZEROBASEDSTRINGS that got introduced in Delphi XE3 and is by default ON in mobile compilers and OFF in Desktop compilers.

Back then, Mark Edington showed a small example of the effects:


procedure ZeroBasedTest;
const
S: string = '012';
begin
{$ZEROBASEDSTRINGS OFF}
Writeln(S[1]); // shows "0"
Writeln(S.Chars[1]); // shows "1"
{$ZEROBASEDSTRINGS ON}
Writeln(S[1]); // shows "1"
Writeln(S.Chars[1]); // shows "1"
end;

view raw

gistfile1.txt

hosted with ❤ by GitHub

and then explained:

The XE3 RTL source code has been refactored to be string index base agnostic. In most cases this is done by utilizing string helper functions which are always zero based.
When it is necessary to traverse a string, the Char[] property is often used to access the individual characters without concern for the current state of the compiler with respect to zero based strings.

In addition, the “Low” and “High” standard functions can now be passed a string variable to provide further flexibility as needed.
When zero based strings are enabled, Low(string) will return 0,  otherwise it will return 1. Likewise, High() returns a bounds adjusted length variation.

The problem is the non-existent forward compatibility of the other compilers (Delphi XE2 and lower).

So if you have library code that needs to work in Delphi versions, you cannot use the High and Low to make the code ZEROBASEDSTRINGS neutral.

Many Delphi developers regularly skip many Delphi versions, so these are still popular:

  • Delphi XE1 and XE2 (the last 2 compilers before Delphi really started to support mobile)
  • Delphi 2007 (the last non-Unicode Delphi compiler)
  • Delphi 7 (the last non-Galileo IDE)

The result is that library code is full of conditionan IF/IFDEF blocks like these:

Fix: this works only in XE3 or higher: “for Index := Low(input) to High(input) do // for ZEROBASEDSTRINGS”


{$ifdef GX_VER240_up}
for Index := Low(input) to High(input) do // for ZEROBASEDSTRINGS
{$else}
for Index := 1 to Length(input) do
{$endif GX_VER240_up}

view raw

gistfile1.txt

hosted with ❤ by GitHub

–jeroen

via: Mark Edington’s Delphi Blog : XE3 RTL Changes: A closer look at TStringHelper.

Posted in Ansi, Delphi, Delphi 2007, Delphi 7, Delphi XE, Delphi XE2, Delphi XE3, Delphi XE4, Delphi XE5, Development, Encoding, Software Development, Unicode | 8 Comments »

Default comparers in Delphi used by TArray.Sort (via: Stack Overflow)

Posted by jpluimers on 2014/11/26

A long while ago, I posted a detailed answer on what functions the default comparers actually were calling to get a feel for if they would apply or not answering delphi – What does the default TArray.Sort comparator actually do and when would you use it? – Stack Overflow.

I needed that information recently because of some sorting issues I bumped into (sorting generic records), so finally a blog post.

First some links to documentation for even more background information:

There is the answer I gave: Read the rest of this entry »

Posted in Algorithms, Ansi, Delphi, Delphi 2009, Delphi 2010, Delphi XE, Delphi XE2, Delphi XE3, Delphi XE4, Delphi XE5, Development, Encoding, Floating point handling, Software Development, Unicode | 2 Comments »

Great answer on “windows – What encoding/code page is cmd.exe using” (via: Stack Overflow)

Posted by jpluimers on 2014/10/06

I just found this [Wayback] great answer (which by now regrettably is deleted; the previous Wayback link still has it) by [Wayback] Јοеу a.k.a. Johannes Rössel on [Wayback] What encoding/code page is cmd.exe using.

The whole answer is worth reading, so I won’t quote only some bits.

Edit 20210609: the answer now has been replaced by an even more detailed answer [Wayback] by [Wayback] andrewdotn. Also recommended reading. The summary of the new answer is this:

The moral of the story?

  • type can print UTF-16LE files with a BOM regardless of your current codepage
  • Win32 programs can be programmed to output Unicode to the console, using WriteConsoleW.
  • Other programs which set the codepage and adjust their output encoding accordingly can print Unicode on the console regardless of what the codepage was when the program started
  • For everything else you will have to mess around with chcp, and will probably still get weird output.

–jeroen

via:   windows – What encoding/code page is cmd.exe using – Stack Overflow.

Posted in Batch-Files, Development, Encoding, Power User, Scripting, Software Development, Windows, Windows 7, Windows 8, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, Windows Vista, Windows XP | Leave a Comment »

Delphi hinting directives: deprecated, experimental, library and platform

Posted by jpluimers on 2014/10/01

I’ve been experimenting with the Delphi hinting directives lately to make it easier to migrate some libraries to newer versions of Delphi and newer platforms.

Hinting directives (deprecated, experimental, library and platform) were – like the $MESSAGE directive – added to Delphi 6.

Up to Delphi 5 you didn’t have any means to declare code obsolete. You had to find clever ways around it.

Warnings for hinting directives

When referring to identifiers marked with a hinting directive, you can get various warning messages that depend on the kind of identifier: unit, or other symbol. Read the rest of this entry »

Posted in Apple Pascal, Borland Pascal, DEC Pascal, Delphi, Delphi 2005, Delphi 2006, Delphi 2007, Delphi 2009, Delphi 2010, Delphi 6, Delphi 7, Delphi 8, Delphi XE, Delphi XE2, Delphi XE3, Delphi XE4, Development, Encoding, FreePascal, ISO-8859, ISO8859, Java, Lazarus, MQ Message Queueing/Queuing, QC, Reflection, Software Development, Sybase, Unicode, UTF-8, UTF8 | 2 Comments »

Windows key character that displays on non-Windows systems (like Mac)

Posted by jpluimers on 2014/08/08

Though there is a Unicode character for the Apple Command Key, there is none for the Windows Key.

The Windows font WinDings does have a character 255 for it, but that font usually is not installed on non-Windows systems. There it will look like Unicode Character ‘LATIN SMALL LETTER Y WITH DIAERESIS’ (U+00FF)

This Unicode code point comes closest to the Windows key: Unicode Character ‘SQUARED PLUS’ (U+229E) and is used by Windows Key page on WikiPedia.

  • The WinDings character looks like this: ÿ
    (non no Windows systems, it will look like an y with two dots on it: ÿ)
  • The Unicode Codepoint U+229E like this: ⊞
    Not a complete match, but pretty close.

The Unicode code points for Mac modifier keys are these:

–jeroen

Posted in Development, Encoding, Mac, Mac OS X / OS X / MacOS, Mac OS X 10.4 Tiger, Mac OS X 10.5 Leopard, Mac OS X 10.6 Snow Leopard, Mac OS X 10.7 Lion, MacBook Retina, MacBook-Air, MacBook-Pro, OS X 10.8 Mountain Lion, Power User, Software Development, Unicode, Windows, Windows 7, Windows 8, Windows Server 2003, Windows Server 2003 R2, Windows Vista, Windows XP, Windows-1252 | Leave a Comment »

Recommended reads when dealing with Character Encodings in software

Posted by jpluimers on 2014/05/06

Apart from the mandatory Joel on Software article about Unicode and Character sets, these two articles are of great value too:

Fun to read from that blog is the Historical Technology  section including this article:

–jeroen

PS: The mandatory one is The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) – Joel on Software.

 

Posted in .NET, Ansi, ASCII, CP437/OEM 437/PC-8, Delphi, Development, EBCDIC, Encoding, ISO-8859, ISO8859, Shift JIS, Software Development, Unicode, UTF-8, UTF8, Windows-1252 | Leave a Comment »

Fiddler2 to the max: inserting proxy authentication to use DropBox (or other app) behind a corporate firewall

Posted by jpluimers on 2014/04/16


A while ago, I was working with a not so cooperative corporate firewall. All web browsers would work fine, but most other applications would not go through the proxy in a nice way.

For instance, DropBox would show the dreadfull “Connection Error” dialog shown on the right.

That dialog basically means “Dropbox has no clue what happens, try fiddling with your proxy or account settings, then press Reconnect Now” to retry.

Many other applications had issues (for instance Visual Studio connecting to Team Foundation System was very unreliable and the workarounds clumsy).

CNTLM: not the solution

I got inspired by the [WayBack] I code and code: Tutorial: How to use Dropbox behind a corporate proxy server using CNTLM, even though I was pretty sure the corporate firewall was not NTLM based.

And indeed, CNTLM -v -M http://google.com -c CNTLM.INI would give errors like this:

cntlm: Proxy returning invalid challenge!
headers_send: fd 4 warning -999 (connection closed)
Connection closed

HTTP Fiddler: looks promising

So I fired up my old buddy [WayBack] Fiddler 2 HTTP debugging proxy.

Further on, you will learn that Fiddler2 is much more, but right now it is enough to know that it basically sits as a local proxy between your applications and the outside world. Read the rest of this entry »

Posted in .NET, .NET 2.0, .NET 3.0, .NET 3.5, .NET 4.0, .NET 4.5, base64, Cntlm, Development, DropBox, Encoding, Fiddler, JavaScript/ECMAScript, NTLM, Power User, Scripting, SocialMedia, Software Development, Web Development, Windows, Windows 7, Windows 8, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, Windows Server 2008 R2, Windows Vista, Windows XP, Windows-Http-Proxy | Leave a Comment »

Cool post from Marc’s Blog: Delphi XE2’s hidden hints and warnings options

Posted by jpluimers on 2014/04/05

A while ago, I had to disable a couple of warnings from legacy code so I could first perform the Unicode conversion, then make time to eliminate the actual warning cause.

This post was much helpful here:

Marc’s Blog: Delphi XE2’s hidden hints and warnings options.

He lists all the W#### and X#### warnings he could find in Delphi XE2 (XE3, XE4 and XE5 more or less have the same), including the mapping to the equivalent directive IDs used inside these blocks:

{$WARN SYMBOL_DEPRECATED ON}
{$WARN SYMBOL_DEPRECATED OFF}
{$WARN SYMBOL_DEPRECATED DEFAULT}
{$WARN SYMBOL_DEPRECATED ERROR}

I also learned that the DEFAULT value restores an option to what you specified in the project settings.

–jeroen

Posted in Delphi, Delphi 2009, Delphi 2010, Delphi XE, Delphi XE2, Delphi XE3, Delphi XE4, Development, Encoding, Software Development, Unicode | 11 Comments »