The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,862 other subscribers

Archive for the ‘Communications Development’ Category

Bash functions to encode and decode the ‘Basic’ HTTP Authentication Scheme

Posted by jpluimers on 2022/02/24

IoT devices still often use the ‘Basic’ HTTP Authentication Scheme for authorisation, see [Wayback] RFC7617: The ‘Basic’ HTTP Authentication Scheme (RFC ) and [Wayback] RFC2617: HTTP Authentication: Basic and Digest Access Authentication (RFC ).

Often this authentication is used even over http instead of over https, for instance the Egardia/Woonveilig alarm devices I wrote about yesterday at  Egardia/Woonveilig: some notes about logging on a local gateway to see more detailed information on the security system. This is contrary to guidance in:

  • RFC7617:
       This scheme is not considered to be a secure method of user
       authentication unless used in conjunction with some external secure
       system such as TLS (Transport Layer Security, [RFC5246]), as the
       user-id and password are passed over the network as cleartext.
  • RFC2617:
       "HTTP/1.0", includes the specification for a Basic Access
       Authentication scheme. This scheme is not considered to be a secure
       method of user authentication (unless used in conjunction with some
       external secure system such as SSL [5]), as the user name and
       password are passed over the network as cleartext.

Fiddling with those alarm devices, I wrote these two little bash functions (with a few notes) that work both on MacOS and in Linux:

# `base64 --decode` is platform neutral (as MacOS uses `-D` and Linux uses `-d`)
# `$1` is the encoded username:password
function decode_http_Basic_Authorization(){
  echo $1 | base64 --decode
  echo
}

# `base64` without parameters encodes
# `echo -n` does not output a new-line
# `$1` is the username; `$2` is the password
function encode_http_Basic_Authorization(){
  echo $1:$2 | base64
}

The first decodes the <credentials> from a Authorization: Basic <credentials> header into a username:password clean text followed by a newline.

The second one encodes a pair of username and password parameters into such a <credentials> string.

They are based on these initial posts that were not cross platform or explanatory:

  1. [Wayback] Decode HTTP Basic Access Authentication – Stack Pointer
  2. [Wayback] Create Authorization Basic Header | MJ’s Web Log

–jeroen

Posted in *nix, *nix-tools, Apple, Authentication, bash, bash, Communications Development, Development, HTTP, Internet protocol suite, Linux, Mac OS X / OS X / MacOS, Power User, Scripting, Security, Software Development, TCP, Web Development | Leave a Comment »

Egardia/Woonveilig: some notes about logging on a local gateway to see more detailed information on the security system

Posted by jpluimers on 2022/02/23

A follow-up on Source: Some links with notes on WoonVeilig/Egardia security system communications, protocols and support by 3rd party home automation apps:

Notes on the Woonveilig/Egardia GATE-03 model alarm hub (where 192.168.x.y is the IPv4 address that hub):

  1. It still uses the plain-text insecure http to communicate, so it is wise to try and put it in a separate LAN apart from other systems.
  2. Logon is done using HTTP Basic access authentication.
  3. Woonveilig/Egardia by now prefers the XMPP prototol over the CID protocol (the CID protocol is still used by jeroenterheerdt/python-egardia.
    • You can find the configuration at http://192.168.x.y/setting/xmpp.htm.
    • XMPP protocol uses
      • xmpp01.egardia.com as primary and xmpp01.alt.egardia.com as secondary server on port 443.
      • arg-####-auth  where ###### are the last 6 *lowercase* hexadecimal digits of the MAC address of the GATE-03.
      • a long password you can find in the plain-text of the http://192.168.x.y/action/xmppGet http GET request fired by http://192.168.x.y/setting/xmpp.htm.
  4. CID protocol address is ip://######@ics.alt.egardia.com:52010/CID where ###### are the last 6 *uppercase* hexadecimal digits of the MAC address of the GATE-03.
  5. User PIN-codes are not visible at the Woonveilig/Egardia alarm site, but they are at http://192.168.x.y/setting/userCode.htm together with their user names.
  6. Special PIN codes for Installer/Duress/Guard/Master/Temporary are at http://192.168.x.y/setting/codeSetting.htm and obtained via http://192.168.x.y/action/areaListGet and http://192.168.x.y/action/codeSettingGet
  7. On the CID protocol:

I got all of the above via: [Wayback/Archive.is] GATE-03 system does not report to Egardiaserver · Issue #26 · jeroenterheerdt/python-egardia (which by coincidence used the same firmware I had: HSGW 0.0.2.18.1 HPGW-L2-XA35H).

Which brings me to some Google search with some remarkable results:

So I did a quick look at LUPUS XT* based products:

Then at the Woonveilig/Egardia and Climax shops:

Conclusions:

  • Egardia/Woonveilig sensors look remarkably similar to the LUPUS ones
  • LUPUS is a re-brand of Climax with slightly different firmware

Side note on open ports

  • Open ports on the [Wayback] GATE-03 alarm device:
    PORT    STATE    SERVICE      VERSION
    9/tcp   filtered discard
    25/tcp  filtered smtp
    80/tcp  open     http
    445/tcp filtered microsoft-ds
  • Open ports on the [Wayback] CAM-06 outdoor camera:
    PORT      STATE    SERVICE      VERSION
    9/tcp     filtered discard
    21/tcp    open     ftp          oftpd
    25/tcp    filtered smtp
    80/tcp    open     tcpwrapped
    445/tcp   filtered microsoft-ds
    554/tcp   open     rtsp
    711/tcp   open     cisco-tdp?
    1935/tcp  open     rtmp?
    6000/tcp  open     X11?
    49152/tcp open     upnp         Portable SDK for UPnP devices 1.6.17 (Linux 3.4.35; UPnP 1.0)
    

jeroen

 

Posted in Communications Development, Development, Python, Scripting, Software Development | Leave a Comment »

In this day and age, web sites with delivery back-ends still have Unicode issues: at least @Woonveilig, @Medireva and @PostNL still have trouble

Posted by jpluimers on 2022/02/09

Nowadays, some 35 years after the first Unicode ideas got drafted and 30+ years after the Unicode Consortium saw the light, UTF-8 is served my more than 95% of the web as shown in yesterday’s post UTF-8 web adoption is huge, closing 100%, but only soured up since around 2006..

I mentioned this:

It means that nowadays there is a very small chance you will see mangled characters (what Japanese call mojibake) when you’re surfing the web.

Serving UTF8 does not mean no unicode problems.

Below are some issues that happened not too long ago and still happen. I have reported them to all parties involved through web-care, but no response whatsoever, and this is bad: Unicode support beyond basic ASCII for the below systems are still broken even for relatively simple non-ASCII characters based in diacritics decorating a standard ASCII character.

Yes, I know the realm of encoding and code pages is a mess, especially when handling data in multiple layers of an application stack. That’s why I wrote this post in the first place, and have a whole encoding category of blog posts plus a Mojibake subset.

Read the rest of this entry »

Posted in Communications Development, CP850, Dark Pattern, Development, Encoding, ISO-8859, ISO8859, Mojibake, Software Development, Unicode, User Experience (ux), UTF-16, UTF-8, Windows-1252 | Leave a Comment »

Some links on Chrome not prompting to save passwords (when Firefox and Safari do)

Posted by jpluimers on 2022/01/20

For quite some time now, Chrome (think years) refuses to prompt for saving passwords whereas Firefox and Safari do prompt and save them, even for site types that it used to save passwords for in the past.

It has been annoying enough for too long now that I tried to do better than the Google searches I used back when I saw this happen first.

Below are some links based on new searches (starting with [Wayback] adding a password in chrome settings – Google Search); hopefully I can try them after I made a list of sites that Chrome does not show the password save prompt for.

Solutions I tried that failed (but maybe useful for others):

Solutions still to try:

Read the rest of this entry »

Posted in Chrome, Chrome, Communications Development, Development, Encryption, ESXi6, ESXi6.5, ESXi6.7, Firefox, Fritz!, Fritz!Box, Fritz!WLAN, Google, https, HTTPS/TLS security, Internet, Internet protocol suite, Let's Encrypt (letsencrypt/certbot), Power User, routers, Safari, Security, TCP, TLS, Virtualization, VMware, VMware ESXi, Web Browsers, Web Development | Leave a Comment »

Which SMTP Port Should I Use? Learn Ports 25, 465, & 587 (and unofficial port 2525) | Mailgun

Posted by jpluimers on 2021/11/30

When trying to deliver mail, it is important to know which protocols and ports you can use.

On smtp, smtp-submission, smtps (ports 25, 587 and 465) and unofficial port 2525 (which Maingun maps to `smtp-submission): [Wayback] Which SMTP Port Should I Use? Learn Ports 25, 465, & 587 | Mailgun

Quote on why smtps port 465 is hardly used:

Port 465:

IANA has reassigned a new service to this port, and it should no longer be used for SMTP communications.

However, because it was once recognized by IANA as valid, there may be legacy systems that are only capable of using this connection method. Typically, you will use this port only if your application demands it. A quick Google search, and you’ll find many consumer Inbox Service Providers’ (ISPs) articles that suggest port 465 as the recommended setup. However, we do not recommend it, as it is not RFC compliant.

–jeroen

Posted in Communications Development, Development, Internet protocol suite, SMTP, Software Development, TLS, Web Development | Leave a Comment »

GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Posted by jpluimers on 2021/11/25

Cool: [Wayback/Archive.is] GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Via: [Archive.is] Jan Schaumann on Twitter: “This looks neat: on-demand remote shell into ephemeral environments, e.g. CI/CD pipeline container. Both sides fetch a client, use rendezvous server to negotiate session info, then establish connection or fall back to proxy through rendezvous. “

Read the rest of this entry »

Posted in Communications Development, Development, DevOps, HTTP, Infrastructure, Internet protocol suite, Power User, Software Development, TCP, WebSockets | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

Random User Generator | Home

Posted by jpluimers on 2021/11/23

Cool tool for when you ever need random users to test a system [Wayback] Random User Generator | Home:

Random user generator is a FREE API for generating placeholder user information. Get profile photos, names, and more. It’s like Lorem Ipsum, for people.

This was used when extracting Parler data to substantiate evidence around the 20210106 USA Capitol riots.

You can even use a simple HTTP GET like [Wayback] randomuser.me/api and get a JSON result like this.

{"results":[{"gender":"female","name":{"title":"Miss","first":"Malou","last":"Mortensen"},"location":{"street":{"number":2669,"name":"Lyngbyvej"},"city":"Sundby","state":"Syddanmark","country":"Denmark","postcode":48047,"coordinates":{"latitude":"-35.1307","longitude":"113.7480"},"timezone":{"offset":"+1:00","description":"Brussels, Copenhagen, Madrid, Paris"}},"email":"malou.mortensen@example.com","login":{"uuid":"981747de-66fe-40b0-87ea-adfe403fe1be","username":"purpleostrich871","password":"sweets","salt":"x86aQbIB","md5":"55497ac53530b428f98b9d36267ceeef","sha1":"358b94ffabe7d827c34da15791e5d6717c594428","sha256":"6e357e887877e29b7e6d53073f648174382c53c24f83479e25fed9c82075ed32"},"dob":{"date":"1995-06-05T04:50:35.145Z","age":26},"registered":{"date":"2018-07-21T00:59:50.523Z","age":3},"phone":"02990797","cell":"94800012","id":{"name":"CPR","value":"050695-9954"},"picture":{"large":"https://randomuser.me/api/portraits/women/27.jpg","medium":"https://randomuser.me/api/portraits/med/women/27.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/27.jpg"},"nat":"DK"}],"info":{"seed":"8971869bb62b73d7","results":1,"page":1,"version":"1.3"}}

Via:

–jeroen

Read the rest of this entry »

Posted in Communications Development, Development, HTTP, Internet protocol suite, JavaScript/ECMAScript, JSON, Python, REST, Scripting, Software Development, TCP | Leave a Comment »

🔎Julia Evans🔍 on Twitter: “ten questions about UDP”

Posted by jpluimers on 2021/11/10

From a while back, but still relevant when you learn all your life:

[Archive.is] 🔎Julia Evans🔍 on Twitter: “ten questions about UDP: “

[WayBack] UDP questions

Hello! Here are some questions & answers. The goal isn’t to get all the questions “right”. Instead, the goal is to learn something! If you find a topic you’re interested in learning more about, I’d encourage you to look it up and learn more

–jeroen

Read the rest of this entry »

Posted in Communications Development, Development, Internet protocol suite, Software Development, UDP | Leave a Comment »

Some links on SMTP tar-pit to lessen SPAM

Posted by jpluimers on 2021/11/03

Some links for my archive; note that pure tar-pits by now are also hampering large email sender services like SendGrid, Mailgun and Amazon SES.

So the below links are for educational and historic purposes only.

I assembled these links because out of a sudden, Ring 2FA verification emails could not be delivered any more.

Ring 2FA came mandatory towards the end of February 2020.

Some links on that:

Sendmail timeouts:

–jeroen

Read the rest of this entry »

Posted in *nix, Communications Development, Development, HIS Host Integration Services, Internet protocol suite, Power User, SMTP | Leave a Comment »