The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

    Mastodon

  • Twitter Updates

  • My Flickr Stream

    20140508-Delphi-2007--Project-Options--Cannot-Edit-Application-Title-HelpFile-Icon-Theming20140430-Fiddler-Filter-Actions-Button-Run-Filterset-now20140424-VMware-Fusion-6.0.3.-no-reclaimable
    More Photos

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,854 other subscribers

Archive for the ‘Development’ Category

« Previous Entries
Next Entries »

What is “deleted” in an information system?

Posted by jpluimers on 2021/11/25

I have had quite a few discussions about data being “deleted” in information systems.

Often, data – despite GDPR – isn’t, or can’t be deleted for many reasons, especially when data is retained on backups, cloud storage is involved or data has been copied in other ways.

Many times, marking with a flag that data is deleted, is enough, but often it isn’t and then you need processes to track down all occurrences of the data and delete it permanently, which can be a tedious job.

Some more interesting thoughts are in this thread that triggered me:

Comment
byu/BlueMountainDace from discussion
inParlerWatch

Posted in Development, Software Development, Systems Architecture | Leave a Comment »

GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Posted by jpluimers on 2021/11/25

Cool: [Wayback/Archive.is] GitHub – TimeToogo/tunshell: Remote shell into ephemeral environments 🐚 🦀

Via: [Archive.is] Jan Schaumann on Twitter: “This looks neat: on-demand remote shell into ephemeral environments, e.g. CI/CD pipeline container. Both sides fetch a client, use rendezvous server to negotiate session info, then establish connection or fall back to proxy through rendezvous. “

Read the rest of this entry »

Posted in Communications Development, Development, DevOps, HTTP, Infrastructure, Internet protocol suite, Power User, Software Development, TCP, WebSockets | Leave a Comment »

Writing desktop apps: use native tools, not web-tools

Posted by jpluimers on 2021/11/24

Despite the Electron framework, you might really want to consider writing desktop applications using native tools as it is extremely hard to write performant desktop applications otherwise.

It isn’t by coincidence that last year, Firefox by default makes the backspace key not go back to the previous web-page: it is still a problem in a truckload of interactive web applications, often even in web-based desktop applications:

I am not alone on this opinion:

In practice, “native” applications based on web-tools are notoriously hard to navigate by keyboard, which essential for swift operation.

I have filed a few bugs, and others many more on this, for example:

Also web-developers tend to love to introduce their own custom UX, like for a 6-digit numeric field, use 6 separate digit fields making it extremely hard to copy/paste numbers.

–jeroen
Read the rest of this entry »

Posted in Development, Software Development, Web Development, Windows Development | Leave a Comment »

OWASP top rated security “feature” A01:2021 – Broken Access Control

Posted by jpluimers on 2021/11/24

An important [Wayback/Archive] A01:2021 – Broken Access Control, in German, is a pre-amble for a future post about getting a feel how to counter the vulnerabilities that OWASP tracks and documents.

Basically remember that Broken Access Control is by far the most vulnerable feature in applications:

Broken Access Control war 2017 auf Platz 5 und ist jetzt Problem #1. 94 % der getesteten Anwendungen hatten irgendeine Form von defekter Zugangskontrolle. Der ehemalige #1 Dauerbrenner Injection ist nur noch auf Platz 3.

Basically the top 3 changed dramatically between 2017 and 2021. The new top-3 is below. Please get acquainted with it.

  1. [Wayback/Archive] A01 Broken Access Control – OWASP Top 10:2021

    Moving up from the fifth position, 94% of applications were tested for some form of broken access control with the average incidence rate of 3.81%, and has the most occurrences in the contributed dataset with over 318k. Notable Common Weakness Enumerations (CWEs) included are CWE-200: Exposure of Sensitive Information to an Unauthorized Actor, CWE-201: Exposure of Sensitive Information Through Sent Data, and CWE-352: Cross-Site Request Forgery.

  2. [Wayback/Archive] A02 Cryptographic Failures – OWASP Top 10:2021
    Shifting up one position to #2, previously known as Sensitive Data Exposure, which is more of a broad symptom rather than a root cause, the focus is on failures related to cryptography (or lack thereof). Which often lead to exposure of sensitive data. Notable Common Weakness Enumerations (CWEs) included are CWE-259: Use of Hard-coded Password, CWE-327: Broken or Risky Crypto Algorithm, and CWE-331 Insufficient Entropy .
  3. [Wayback/Archive] A03 Injection – OWASP Top 10:2021

    Injection slides down to the third position. 94% of the applications were tested for some form of injection with a max incidence rate of 19%, an average incidence rate of 3%, and 274k occurances. Notable Common Weakness Enumerations (CWEs) included are CWE-79: Cross-site Scripting, CWE-89: SQL Injection, and CWE-73: External Control of File Name or Path.

Via; [Archive] Kristian Köhntopp on Twitter: “Vieles aus diesem Thread ist nun geordneter in … zu finden.… “

Very much related as A01 was the basic cause of GitHub’s commitment to npm ecosystem security | The GitHub Blog – no npm package can historically ben tracked to be authentic.

We determined that this vulnerability was due to inconsistent authorization checks and validation of data across several microservices that handle requests to the npm registry. In this architecture, the authorization service was properly validating user authorization to packages based on data passed in request URL paths. However, the service that performs underlying updates to the registry data determined which package to publish based on the contents of the uploaded package file.

–jeroen

Posted in Development, Power User, Security, Software Development | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

view raw

Makefile.postfix

hosted with ❤ by GitHub

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick Koetter, Ralf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

Random User Generator | Home

Posted by jpluimers on 2021/11/23

Cool tool for when you ever need random users to test a system [Wayback] Random User Generator | Home:

Random user generator is a FREE API for generating placeholder user information. Get profile photos, names, and more. It’s like Lorem Ipsum, for people.

This was used when extracting Parler data to substantiate evidence around the 20210106 USA Capitol riots.

You can even use a simple HTTP GET like [Wayback] randomuser.me/api and get a JSON result like this.

{"results":[{"gender":"female","name":{"title":"Miss","first":"Malou","last":"Mortensen"},"location":{"street":{"number":2669,"name":"Lyngbyvej"},"city":"Sundby","state":"Syddanmark","country":"Denmark","postcode":48047,"coordinates":{"latitude":"-35.1307","longitude":"113.7480"},"timezone":{"offset":"+1:00","description":"Brussels, Copenhagen, Madrid, Paris"}},"email":"malou.mortensen@example.com","login":{"uuid":"981747de-66fe-40b0-87ea-adfe403fe1be","username":"purpleostrich871","password":"sweets","salt":"x86aQbIB","md5":"55497ac53530b428f98b9d36267ceeef","sha1":"358b94ffabe7d827c34da15791e5d6717c594428","sha256":"6e357e887877e29b7e6d53073f648174382c53c24f83479e25fed9c82075ed32"},"dob":{"date":"1995-06-05T04:50:35.145Z","age":26},"registered":{"date":"2018-07-21T00:59:50.523Z","age":3},"phone":"02990797","cell":"94800012","id":{"name":"CPR","value":"050695-9954"},"picture":{"large":"https://randomuser.me/api/portraits/women/27.jpg","medium":"https://randomuser.me/api/portraits/med/women/27.jpg","thumbnail":"https://randomuser.me/api/portraits/thumb/women/27.jpg"},"nat":"DK"}],"info":{"seed":"8971869bb62b73d7","results":1,"page":1,"version":"1.3"}}

Via:

–jeroen

Read the rest of this entry »

Posted in Communications Development, Development, HTTP, Internet protocol suite, JavaScript/ECMAScript, JSON, Python, REST, Scripting, Software Development, TCP | Leave a Comment »

IFTTT alternatives

Posted by jpluimers on 2021/11/23

A long time ago, I experimented a bit with if this then that, but found it too limited for my purposes. Though there were many integrations, the level of automation wsa very limited, especially in the kinds and flexibility of rules. Nice idea, not well executed.

By now, they even charge for it.

So here are some alternatives you might want to use:

Via [Wayback] Ifttt introduceert betaald Pro-abonnement en beperkt gratis versie – Beeld en geluid – Nieuws – Tweakers

–jeroen

Posted in Development, Power User, Software Development | Leave a Comment »

Tool for debugging makefiles – Stack Overflow

Posted by jpluimers on 2021/11/23

I’m not good at makefiles, so I needed a tool to help me debug some relatively simple ones.

[Wayback] Tool for debugging makefiles – Stack Overflow came to the rescue.

The suggestion in the first answer was enough: run make -n.

But the other suggestions are great too, so here I quote them (thanks [Wayback] Rob Wells and [Wayback] User Rajish):

Have you been looking at the output from running make -n and make -np, and the biggie make -nd?

Are you using a fairly recent version of gmake?

Have you looked at the free chapter on [Wayback] Debugging Makefiles available on O’Reilly’s site for their excellent book “Managing Projects with GNU Make” ([Wayback] Amazon Link).

I’m sure that [Wayback] remake is what you are looking for.

From the homepage:

remake is a patched and modernized version of GNU make utility that adds improved error reporting, the ability to trace execution in a comprehensible way, and a debugger.

It has gdb-like interface and is supported by mdb-mode in (x)emacs which means breakponts, watches etc. And there’s [Wayback] DDD if you don’t like (x)emacs

The free chapter: [Wayback] https://www.oreilly.com/openbook/make3/book/ch12.pdf

Remake documentation: [Wayback] remake – GNU Make with comprehensible tracing, profiling, extended error messages, and a debugger — remake 4.3+dbg-1.4 documentation

Tutorials

[Wayback] Makefile Tutorial by Example

–jeroen

Posted in Development, Makefile, Scripting, Software Development | Leave a Comment »

One needs to actually fail in order to get a feel for reality

Posted by jpluimers on 2021/11/18

Failure is very important to understand about reality.

Software Development and DevOps should do this more often, as real world failures are always different from test cases.

This is especially true in the field of web sites, cloud development, kubernetes, microservices and other distributed “Stuff” (which starts with DNS!).

There was a great thread last year that ended with this quote:

“Error Budget”: How much infrastructure you’re allowed to set on fire to learn the meaning of the word “heiß”. Every organization has an error budget, but most don’t plan for it.

The start of the thread is around [Archive.is] Kristian Köhntopp on Twitter: “So the little one was a bit over 12 months old, and could already say “Mama” and “Papa”. It was around christmas, and there was a candle on the table, glowing interestingly, so he wanted to touch it. Of course I told him “Nein, heiß!”.”.

Then Kristian Köhntopp summarised the thread in this great blog post: [Wayback] On Touching Candles, And Error Budgets | Die wunderbare Welt von Isotopp.

Too bad ThreadReaderApp still is unable to archive trees of messages (not even single threaded trees with multiple participants) as this is possible with the Twitter v2 API: [Archive.is] twopcharts_nl on Twitter: “is wel eenvoudig mogelijk nu met api v2.… “

So I archived the thread in two Archive.is links:

In the thread, Kristian also mentions [Wayback] Code rant: The Configuration Complexity Clock

That article has a very important observation:

At a certain level of complexity, hard-coding a solution may be the least evil option.

–jeroen

Read the rest of this entry »

Posted in Cloud Development, Development, Micro Services, Software Development | Leave a Comment »

GitHub’s commitment to npm ecosystem security | The GitHub Blog – no npm package can historically ben tracked to be authentic

Posted by jpluimers on 2021/11/18

What started as [Wayback/Archive] GitHub’s commitment to npm ecosystem security | The GitHub Blog with this qoute:

on November 2 we received a report to our [Wayback/Archive] security bug bounty program of a vulnerability that would allow an attacker to publish new versions of any npm package using an account without proper authorization. …

This vulnerability existed in the npm registry beyond the timeframe for which we have telemetry to determine whether it has ever been exploited maliciously. … the timeframe for which we have available telemetry, which goes back to September 2020 …

This basically means you cannot trust the authenticity of any npm package published through the registry.

It sparked reactions like these:

–jeroen

Posted in Development, JavaScript/ECMAScript, Node.js, Scripting, Software Development | Leave a Comment »

« Previous Entries
Next Entries »