Archive for the ‘Encoding’ Category
Posted by jpluimers on 2024/08/08
I responded to [Wayback/Archive] jilles.com on Twitter: “@0xD4ni @Twitter What is the regexp for an emoticon ?” with [Wayback/Archive] Jeroen Wiert Pluimers on Twitter: “@jilles_com @0xD4ni @Twitter \p{So}+ See …”.
I got the answer from [Wayback/Archive] java – What is the regex to extract all the emojis from a string? – Stack Overflow (thanks [Wayback/Archive] vishalaksh, and [Wayback/Archive] Desgard_Duan) which refers to the quoted section below.
Note that correctly matching highly depends on the versions of the libraries you use: there have been lots of releases of Unicode versions over the last years (since 2014 roughly every 12 months) each usually adding more Emoji.
In addition, many Emoji are not single Unicode codepoints: often they are code points (with or without any of the variation selectors) stacked on top of each other with zero-width joiners like I described in Kris on Twitter: “Company chat: »Right, we need more languages with Emoji as variable type indicators and pointer symbols.«….
I tried fiddling on [Wayback/Archive] regex101: build, test, and debug regex and could not always getting it to work as I hoped for, but also could not figure out how recent their libraries are.
Read the rest of this entry »
Posted in Conference Topics, Conferences, Development, Emoticons, Encoding, Event, Geeky, RegEx, Software Development, Unicode | Leave a Comment »
Posted by jpluimers on 2024/08/07
At the time of writing a lot of this might be more recent, but for quite some time codepoints.net had not been updated with code point information newer Unicode releases.
Basically it was stuck at Unicode version 8.0 with some 120k glyphs. At the time of writing Unicode version 15.0 is in beta and the difference between 15.0 and 8.0 is some 24k glyphs.
So I had a quick twitter chat with the author and jotted down the links in this blog post so I won’t forget them.
There I learned it was open source (I think it is the only Unicode codepoint site that is).
Here it goes:
Read the rest of this entry »
Posted in *nix, *nix-tools, Apache2, codepoints.net, Conference Topics, Conferences, Database Development, Debian, Development, DVCS - Distributed Version Control, Encoding, Event, GitHub, Linux, MySQL, PHP, Power User, Scripting, Software Development, Source Code Management, Unicode, Web Development | Leave a Comment »
Posted by jpluimers on 2024/08/06
Please do not overdo Unicode outside the ASCII realm for identifiers and stay away from Emoji: [Wayback/Archive] Kris on Twitter: “Company chat: »Right, we need more languages with Emoji as variable type indicators and pointer symbols.«…”
Company chat: »Right, we need more languages with Emoji as variable type indicators and pointer symbols.«
»🎼initializer🎱«
»💦 mutable, 🧱 not.«
»🎁 on the heap, 🥞 on the stack«
»🍼 ctor, 🪦 dtor«
»� non-utf string result«
»any of 👩❤️💋👨 as a concat operator«
»📁📂 block delims«
Read the rest of this entry »
Posted in Conference Topics, Conferences, Development, Encoding, Event, Fun, Quotes, Software Development, Unicode | Leave a Comment »
Posted by jpluimers on 2023/05/25
A while ago, I learned that MacOS has had a Unicode Hex Input keyboard since ages.
It is not installed by default, so you have to manually add it:
- Start the
System Preferences.app
- Open the
Keyboard icon
- Choose the
Input Sources tab
- Click the plus (
+) icon
- Search for Unicode or Hex to get so
Unicode Hex Input is the only entry in the list
- Click the
Add button
- Choose the
Keyboard tab
- Enable Show keyboard and emoji viewers in menu bar
Now in the menu bar, you can select the Unicode Hex Input.
After that, when holding the Option key, any 4-digit Unicode sequence will get you a Unicode character.
Read the rest of this entry »
Posted in Apple, Development, Encoding, Mac OS X / OS X / MacOS, Power User, Software Development, Unicode | Leave a Comment »
Posted by jpluimers on 2022/11/22
I have been into more and more Mojibake example pages like [Wayback] Mojibake: Question Marks, Strange Characters and Other Issues | GPI
Have you ever found strange characters like these ��� when viewing content in applications or websites in other languages?
They made me realise that all these (including the Mojibake examples on my blog) are just artifacts, but the real list of examples is the set of ftfy test cases at [Wayback/Archive.is] python-ftfy/test_cases.json at master · LuminosoInsight/python-ftfy
I got reminded when Waternet moved from paper mail using “Pyreneeën” to email using “Pyreneeën“. Not as bad as Waterschap AGV did earlier: they took it one level further and made “Pyreneeën” out of it, see Last year, a classic Mojibake was introduced when Waterschap Amstel, Gooi en Vecht redesigned their IT systems.
This seems like a trend where newer systems perform worse than older systems. I wonder why that is.
BTW: the trick on the [Wayback/Archive] Python.org shell to run ftfy (which is not installed by default) is first dropping to the shell (see my post How do I drop a bash shell from within Python? – Stack Overflow), then starting python again:
Read the rest of this entry »
Posted in CP850, Development, Encoding, ftfy, ISO-8859, Mojibake, Python, Scripting, Software Development, Unicode, UTF-8, UTF8 | Leave a Comment »
Posted by jpluimers on 2022/09/01
Geocities is long dead, but luckily a lot has been archived: [Wayback] Archive.is: History of ASCII Art with a very comprehensive history ranging from ancient old hand painted art to contemporary computer made illustrations.
Via: [Wayback/Archive.is] ASCII art: The roots of ASCII art
--jeroen
Posted in ASCII, ASCII art / AsciiArt, Development, Encoding, Fun, History, Power User, Retrocomputing, Software Development | Leave a Comment »
Posted by jpluimers on 2022/07/06
Early june, I blogged about Wake-on-LAN from a Windows machine.
My plan was to adopt [Wayback/Archive.is] Wake.ps1 into Wake-on-LAN.ps1 (as naming is important).
One of the goals was to support multiple hardware MAC address formats, especially as Wake.ps1 had the below comment, but did support the AA-BB-CC-DD-EE-FF, though not the AA:BB:CC:DD:EE:FF hardware MAC address format:
<#
...
.NOTES
Make sure the MAC addresses supplied don't contain "-" or ".".
#>
A colon separated hardware MAC address would result in this error inside the call to the [Wayback/Archive.is] PhysicalAddress.Parse Method (System.Net.NetworkInformation) | Microsoft Docs:
Send-Packet : Exception calling "Parse" with "1" argument(s): "An invalid physical address was specified."
So I did some digging, starting inside the above mentioned blog post, and adding more:
- Wake.ps1 uses the [Wayback/Archive.is]
Parse method in the [Wayback/Archive.is] PhysicalAddress.cs source code in C# .NET, which contains code like this:
//has dashes?
if (address.IndexOf('-') >= 0 ){
hasDashes = true;
buffer = new byte[(address.Length+1)/3];
}
- The Perl script at [Wayback/Archive.is] wakeonlan/wakeonlan at master · jpoliv/wakeonlan that started my first blog post in this series which mentions:
xx:xx:xx:xx:xx:xx (canonical)xx-xx-xx-xx-xx-xx (Windows)xxxxxx-xxxxxx (Hewlett-Packard switches)xxxxxxxxxxxx (Intel Landesk)
I should rename the first one IEEE 802, as per this:
- The MAC address: Notational conventions – Wikipedia
The standard (IEEE 802) format for printing EUI-48 addresses in human-friendly form is six groups of two hexadecimal digits, separated by hyphens (-) in transmission order (e.g. 01-23-45-67-89-AB). This form is also commonly used for EUI-64 (e.g. 01-23-45-67-89-AB-CD-EF).[2] Other conventions include six groups of two hexadecimal digits separated by colons (:) (e.g. 01:23:45:67:89:AB), and three groups of four hexadecimal digits separated by dots (.) (e.g. 0123.4567.89AB); again in transmission order.[30]
The latter is used by Cisco (see for instance [Wayback/Archive.is] Cisco DCNM Security Configuration Guide, Release 4.0 – Configuring MAC ACLs [Support] – Cisco and [Wayback/Archive.is] Cisco IOS LAN Switching Command Reference – mac address-group through revision [Support] – Cisco), so another format to add:
- [Wayback/Archive.is] PhysicalAddress.Parse Method (System.Net.NetworkInformation) | Microsoft Docs remarks:
The address parameter must contain a string that can only consist of numbers and letters as hexadecimal digits. Some examples of string formats that are acceptable are as follows:
00112233445500-11-22-33-44-550011.2233.445500:11:22:33:44:55F0-E1-D2-C3-B4-A5f0-e1-d2-c3-b4-a5
Use the GetAddressBytes method to retrieve the address from an existing PhysicalAddress instance.
- After a bit more digging via [Wayback/Archive.is] “three groups of four hexadecimal digits separated by dots” – Google Search , I found that even more hardware MAC address formats are in use as per [Wayback/Archive.is] What are the various standard and industry practice ways to express a 48-bit MAC address? – Network Engineering Stack Exchange.
I really do not have all the sources for the various representations for 48-bit MAC addresses, but I have seen them variously used:
AA-BB-CC-DD-EE-FF
AA.BB.CC.DD.EE.FF
AA:BB:CC:DD:EE:FF
AAA-BBB-CCC-DDD
AAA.BBB.CCC.DDD
AAA:BBB:CCC:DDD
AAAA-BBBB-CCCC
AAAA.BBBB.CCCC
AAAA:BBBB:CCCC
AAAAAA-BBBBBB
AAAAAA.BBBBBB
AAAAAA:BBBBBB
From the last list, which is far more complete than the others, I recognise quite a few from tools I used in the past, but too forgot the actual sources, so I took the full list from there and tried to name them in parenthesis after the links I found above and what I remembered:
AABBCCDDEEFF (Bare / Landesk)AA-BB-CC-DD-EE-FF (IEEE 802 / Windows)AA.BB.CC.DD.EE.FF (???)AA:BB:CC:DD:EE:FF (Linux / BSD / MacOS)AAA-BBB-CCC-DDD (???)AAA.BBB.CCC.DDD (Cisco?)AAA:BBB:CCC:DDD (???)AAAA-BBBB-CCCC (???)AAAA.BBBB.CCCC (Cisco / Brocade)AAAA:BBBB:CCCC (???)AAAAAA-BBBBBB (Hewlett-Packard networking)AAAAAA.BBBBBB (???)AAAAAA:BBBBBB (???)
Some additional links in addition to the ones above:
–jeroen
Posted in .NET, CommandLine, Development, Encoding, HEX encoding, Network-and-equipment, Power User, PowerShell, PowerShell, Scripting, Software Development | Leave a Comment »
Posted by jpluimers on 2022/06/30
Even with a batch file saved as UTF-8 (with or without BOM), by default it does not show most non-ASCII Unicode characters.
The reason is that the default codepage usually is an ANSI one like codepage 437.
Thanks [Wayback] niutech for answering [Wayback/Archive.is] Unicode symbols in a batch file – Stack Overflow:
You can manually set the codepage to UTF-8 by typing chcp 65001 at the top of your batch file.
Codepage 65001 is Windows speak for the UTF-8 code page. I have some more blog entries mentioning codepage 65001.
An example where I needed this was to show how to address the localghost from a batch file (see The spookback localghost address to resolve 👻). This was the resulting UTF-8 saved batch file:
chcp 65001
ping 👻
ping xn--9q8h
For single-byte non-ASCII characters, you can usually get away with setting the encoding of your batch file to your default code page as mentioned in [Wayback/Archive.is] cmd – Using box-drawing Unicode characters in batch files – Stack Overflow.
–jeroen
Posted in Batch-Files, Development, Encoding, Scripting, Software Development, Unicode, UTF-8, Windows Development | Leave a Comment »
Posted by jpluimers on 2022/06/23
[Wayback/Archive.is] CyberChef:
a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser. These operations include simple encoding like XOR or Base64, more complex encryption like AES, DES and Blowfish, creating binary and hexdumps, compression and decompression of data, calculating hashes and checksums, IPv6 and X.509 parsing, changing character encodings, and much more.
Source code at [Wayback/Archive.is] gchq/CyberChef: The Cyber Swiss Army Knife – a web app for encryption, encoding, compression and data analysis.
Via [Archive.is] Jilles🏳️🌈 on Twitter: “Hidden in plain sight. Rot13 cross word. Hidden Barcodes. Qr codes. Barely any InfoSec skill required. Still a hand full. Usually my to go place is: Cyberchef. I did a fun one for cyberklaas using ansi art.… “
Jilles also pointed to the solving part in [Archive.is] Jilles🏳️🌈 on Twitter: “See also, for solving: SCWF… “
The [Wayback/Archive.is] Solve Crypto with Force! needs to run without most script blockers, so best run it in an anonymous/private browser window.
Source code for SCWF is at [Wayback/Archive.is] DaWouw/SCWF: CTF tool for identifying, brute forcing and decoding encryption schemes in an automated way.
Screen shot of Cyberchef example “Perform AES decryption, extracting the IV from the beginning of the cipher stream” [Archive.is]:
Read the rest of this entry »
Posted in Barcode, Cyberchef, Development, EAN, Encoding, Encryption, Hashing, Power User, QR code, Security, Software Development | Leave a Comment »
The Wiert Corner - irregular stream of stuff 