The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,860 other subscribers

Archive for November, 2021

What is “deleted” in an information system?

Posted by jpluimers on 2021/11/25

I have had quite a few discussions about data being “deleted” in information systems.

Often, data – despite GDPR – isn’t, or can’t be deleted for many reasons, especially when data is retained on backups, cloud storage is involved or data has been copied in other ways.

Many times, marking with a flag that data is deleted, is enough, but often it isn’t and then you need processes to track down all occurrences of the data and delete it permanently, which can be a tedious job.

Some more interesting thoughts are in this thread that triggered me:

Comment
byu/BlueMountainDace from discussion
inParlerWatch

Posted in Development, Software Development, Systems Architecture | Leave a Comment »

GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Posted by jpluimers on 2021/11/25

Cool: [Wayback/Archive.is] GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Via: [Archive.is] Jan Schaumann on Twitter: “This looks neat: on-demand remote shell into ephemeral environments, e.g. CI/CD pipeline container. Both sides fetch a client, use rendezvous server to negotiate session info, then establish connection or fall back to proxy through rendezvous. “

Read the rest of this entry »

Posted in Communications Development, Development, DevOps, HTTP, Infrastructure, Internet protocol suite, Power User, Software Development, TCP, WebSockets | Leave a Comment »

Writing desktop apps: use native tools, not web-tools

Posted by jpluimers on 2021/11/24

Despite the Electron framework, you might really want to consider writing desktop applications using native tools as it is extremely hard to write performant desktop applications otherwise.

It isn’t by coincidence that last year, Firefox by default makes the backspace key not go back to the previous web-page: it is still a problem in a truckload of interactive web applications, often even in web-based desktop applications:

I am not alone on this opinion:

In practice, “native” applications based on web-tools are notoriously hard to navigate by keyboard, which essential for swift operation.

I have filed a few bugs, and others many more on this, for example:

Also web-developers tend to love to introduce their own custom UX, like for a 6-digit numeric field, use 6 separate digit fields making it extremely hard to copy/paste numbers.

–jeroen
Read the rest of this entry »

Posted in Development, Software Development, Web Development, Windows Development | Leave a Comment »

OWASP top rated security “feature” A01:2021 – Broken Access Control

Posted by jpluimers on 2021/11/24

An important [Wayback/Archive] A01:2021 – Broken Access Control, in German, is a pre-amble for a future post about getting a feel how to counter the vulnerabilities that OWASP tracks and documents.

Basically remember that Broken Access Control is by far the most vulnerable feature in applications:

Broken Access Control war 2017 auf Platz 5 und ist jetzt Problem . 94 % der getesteten Anwendungen hatten irgendeine Form von defekter Zugangskontrolle. Der ehemalige Dauerbrenner Injection ist nur noch auf Platz 3.

Basically the top 3 changed dramatically between 2017 and 2021. The new top-3 is below. Please get acquainted with it.

  1. [Wayback/Archive] A01 Broken Access Control – OWASP Top 10:2021

    Moving up from the fifth position, 94% of applications were tested for some form of broken access control with the average incidence rate of 3.81%, and has the most occurrences in the contributed dataset with over 318k. Notable Common Weakness Enumerations (CWEs) included are CWE-200: Exposure of Sensitive Information to an Unauthorized ActorCWE-201: Exposure of Sensitive Information Through Sent Data, and CWE-352: Cross-Site Request Forgery.

  2. [Wayback/Archive] A02 Cryptographic Failures – OWASP Top 10:2021
    Shifting up one position to , previously known as Sensitive Data Exposure, which is more of a broad symptom rather than a root cause, the focus is on failures related to cryptography (or lack thereof). Which often lead to exposure of sensitive data. Notable Common Weakness Enumerations (CWEs) included are CWE-259: Use of Hard-coded PasswordCWE-327: Broken or Risky Crypto Algorithm, and CWE-331 Insufficient Entropy .
  3. [Wayback/Archive] A03 Injection – OWASP Top 10:2021

    Injection slides down to the third position. 94% of the applications were tested for some form of injection with a max incidence rate of 19%, an average incidence rate of 3%, and 274k occurances. Notable Common Weakness Enumerations (CWEs) included are CWE-79: Cross-site ScriptingCWE-89: SQL Injection, and CWE-73: External Control of File Name or Path.

Via; [Archive] Kristian Köhntopp on Twitter: “Vieles aus diesem Thread ist nun geordneter in … zu finden.… “

Very much related as A01 was the basic cause of GitHub’s commitment to npm ecosystem security | The GitHub Blog – no npm package can historically ben tracked to be authentic.

We determined that this vulnerability was due to inconsistent authorization checks and validation of data across several microservices that handle requests to the npm registry. In this architecture, the authorization service was properly validating user authorization to packages based on data passed in request URL paths. However, the service that performs underlying updates to the registry data determined which package to publish based on the contents of the uploaded package file.

–jeroen

Posted in Development, Power User, Security, Software Development | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

Random User Generator | Home

Posted by jpluimers on 2021/11/23

Cool tool for when you ever need random users to test a system [Wayback] Random User Generator | Home:

Random user generator is a FREE API for generating placeholder user information. Get profile photos, names, and more. It’s like Lorem Ipsum, for people.

This was used when extracting Parler data to substantiate evidence around the 20210106 USA Capitol riots.

You can even use a simple HTTP GET like [Wayback] randomuser.me/api and get a JSON result like this.

{"results":[{"gender":"female","name":{"title":"Miss","first":"Malou","last":"Mortensen"},"location":{"street":{"number":2669,"name":"Lyngbyvej"},"city":"Sundby","state":"Syddanmark","country":"Denmark","postcode":48047,"coordinates":{"latitude":"-35.1307","longitude":"113.7480"},"timezone":{"offset":"+1:00","description":"Brussels, Copenhagen, Madrid, Paris"}},"email":"malou.mortensen@example.com","login":{"uuid":"981747de-66fe-40b0-87ea-adfe403fe1be","username":"purpleostrich871","password":"sweets","salt":"x86aQbIB","md5":"55497ac53530b428f98b9d36267ceeef","sha1":"358b94ffabe7d827c34da15791e5d6717c594428","sha256":"6e357e887877e29b7e6d53073f648174382c53c24f83479e25fed9c82075ed32"},"dob":{"date":"1995-06-05T04:50:35.145Z","age":26},"registered":{"date":"2018-07-21T00:59:50.523Z","age":3},"phone":"02990797","cell":"94800012","id":{"name":"CPR","value":"050695-9954"},"picture":{"large":"https://randomuser.me/api/portraits/women/27.jpg","medium":"https://randomuser.me/api/portraits/med/women/27.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/27.jpg"},"nat":"DK"}],"info":{"seed":"8971869bb62b73d7","results":1,"page":1,"version":"1.3"}}

Via:

–jeroen

Read the rest of this entry »

Posted in Communications Development, Development, HTTP, Internet protocol suite, JavaScript/ECMAScript, JSON, Python, REST, Scripting, Software Development, TCP | Leave a Comment »

IFTTT alternatives

Posted by jpluimers on 2021/11/23

A long time ago, I experimented a bit with if this then that, but found it too limited for my purposes. Though there were many integrations, the level of automation wsa very limited, especially in the kinds and flexibility of rules. Nice idea, not well executed.

By now, they even charge for it.

So here are some alternatives you might want to use:

Via [Wayback] Ifttt introduceert betaald Pro-abonnement en beperkt gratis versie – Beeld en geluid – Nieuws – Tweakers

–jeroen

Posted in Development, Power User, Software Development | Leave a Comment »

Tool for debugging makefiles – Stack Overflow

Posted by jpluimers on 2021/11/23

I’m not good at makefiles, so I needed a tool to help me debug some relatively simple ones.

[Wayback] Tool for debugging makefiles – Stack Overflow came to the rescue.

The suggestion in the first answer was enough: run make -n.

But the other suggestions are great too, so here I quote them (thanks [Wayback] Rob Wells and [Wayback] User Rajish):

Have you been looking at the output from running make -n and make -np, and the biggie make -nd?

Are you using a fairly recent version of gmake?

Have you looked at the free chapter on [Wayback] Debugging Makefiles available on O’Reilly’s site for their excellent book “Managing Projects with GNU Make” ([Wayback] Amazon Link).


I’m sure that [Waybackremake is what you are looking for.

From the homepage:

remake is a patched and modernized version of GNU make utility that adds improved error reporting, the ability to trace execution in a comprehensible way, and a debugger.

It has gdb-like interface and is supported by mdb-mode in (x)emacs which means breakponts, watches etc. And there’s [Wayback] DDD if you don’t like (x)emacs

The free chapter: [Wayback] https://www.oreilly.com/openbook/make3/book/ch12.pdf

Remake documentation: [Wayback] remake – GNU Make with comprehensible tracing, profiling, extended error messages, and a debugger — remake 4.3+dbg-1.4 documentation

Tutorials

[Wayback] Makefile Tutorial by Example

–jeroen

Posted in Development, Makefile, Scripting, Software Development | Leave a Comment »

SNPG – Aanmelden patiënten met een afweerstoornis voor derde prik tegen het Covid-19 coronavirus

Posted by jpluimers on 2021/11/22

[Wayback/Archive] SNPG – Aanmelden patiënten met een afweerstoornis voor derde prik

Om een patiënt door te verwijzen kunt u een uitnodigingsbrief aanmaken door het volgende formulier in te vullen: [Wayback/Archive] www.formdesk.nl/rivm2/immuno2021. Als u de naam, het adres, woonplaats en emailadres van de patiënt invult in wordt er een uitnodigingsbrief aangemaakt. Dit gebeurt onder AVG-voorwaarden, het RIVM heeft geen toegang tot deze gegevens.

Via [Archive] Valerie van de Flier on Twitter: “Hier kunnen artsen het aanmeldingsformulier vinden: … “

–jeroen

Posted in LifeHacker, Power User | Leave a Comment »

PiKVM v3 HAT by Maxim Devaev » Shipping in progress — Kickstarter

Posted by jpluimers on 2021/11/22

Oh cool: should arrive during the holiday season:

~500pc have been shipped to the backers in the USA and CanadaWe are planning to ship ALL pledges that don’t contain the case by Nov, 19.We are planning to ship ALL pledges that include the case by Dec, 3.Depending on your location it might take from a couple of days to a couple of weeks to actually receive the shipments.Important: we send a notification with a tracking number for every shipment. Please check your Spam folder from time to time.

Source: [Wayback/Archive] PiKVM v3 HAT by Maxim Devaev » Shipping in progress — Kickstarter

Documentation is at [Wayback/Archive] PiKVM v3 HAT guide – PiKVM Handbook via [Wayback/Archive] Thank you for buying PiKVM v3 HAT! showing that the ATX case bracket needs to be 3D-printed separately:

Read the rest of this entry »

Posted in Hardware, KVM keyboard/video/mouse, PiKVM / Pi-KVM, Power User | Leave a Comment »