The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,815 other followers

Archive for the ‘postfix’ Category

Some links on rate-limiting Postfix

Posted by jpluimers on 2022/04/11

For my link archive:

 

The crux seems to be a combination of these parameters to do outgoing rate-limiting, for instance with these values:

default_destination_concurrency_limit = 30
default_destination_rate_delay = 5s

Here default_destination_concurrency_limit needs to be larger than one (so mails are grouped by domain) in order for the default_destination_rate_delay to have any effect at all.

You can even extend this to named transports as per [Wayback] Postfix Users – Create Custom Mail Queue answered by [Wayback] Ralf Hildebrandt

> Can I create custom mail queue in /var/spool/postfix to hold the mails for
> specific destination and schedule to deliver one by one for period of time,
> let’s say 2 mins.

That’s not needed. Create a custom transport for the destination.
Then use
<nameofcustomtransport>_destination_rate_delay = 120s

 

In the below bullets from the Postfix documentation, emphasis is mine.

  • [Wayback] Postfix Configuration Parameters: main.cf
    • [Wayback] Postfix Configuration Parameters: default_destination_concurrency_limit

      default_destination_concurrency_limit (default: 20)

      The default maximal number of parallel deliveries to the same destination. This is the default limit for delivery via the lmtp(8)pipe(8)smtp(8) and virtual(8) delivery agents. With per-destination recipient limit > 1, a destination is a domain, otherwise it is a recipient.

      Use transport_destination_concurrency_limit to specify a transport-specific override, where transport is the master.cf name of the message delivery transport.

    • [Wayback] Postfix Configuration Parameters: default_destination_rate_delay

      default_destination_rate_delay (default: 0s)

      The default amount of delay that is inserted between individual message deliveries to the same destination and over the same message delivery transport. Specify a non-zero value to rate-limit those message deliveries to at most one per $default_destination_rate_delay.

      The resulting behavior depends on the value of the corresponding per-destination recipient limit.

      • With a corresponding per-destination recipient limit > 1, the rate delay specifies the time between deliveries to the same domain. Different domains are delivered in parallel, subject to the process limits specified in master.cf.
      • With a corresponding per-destination recipient limit equal to 1, the rate delay specifies the time between deliveries to the same recipient. Different recipients are delivered in parallel, subject to the process limits specified in master.cf.

      To enable the delay, specify a non-zero time value (an integral value plus an optional one-letter suffix that specifies the time unit).

      Time units: s (seconds), m (minutes), h (hours), d (days), w (weeks). The default time unit is s (seconds).

      NOTE: the delay is enforced by the queue manager. The delay timer state does not survive “postfix reload” or “postfix stop“.

      Use transport_destination_rate_delay to specify a transport-specific override, where transport is the master.cf name of the message delivery transport.

      NOTE: with a non-zero _destination_rate_delay, specify a transport_destination_concurrency_failed_cohort_limit of 10 or more to prevent Postfix from deferring all mail for the same destination after only one connection or handshake error.

      This feature is available in Postfix 2.5 and later.

    • [Wayback] Postfix Configuration Parameters: default_process_limit

      default_process_limit (default: 100)

      The default maximal number of Postfix child processes that provide a given service. This limit can be overruled for specific services in the master.cf file.

  • [Wayback] Postfix manual – master(5): master.cf (where Process limit is the 7th parameter on a configuration line)

           Process limit (default: $default_process_limit)
                  The maximum number of processes that may  execute  this  service
                  simultaneously. Specify 0 for no process count limit.
    
                  NOTE:  Some  Postfix  services  must  be  configured  as  a sin-
                  gle-process service (for example,  qmgr(8))  and  some  services
                  must   be   configured  with  no  process  limit  (for  example,
                  cleanup(8)).  These limits must not be changed.

–jeroen

 

 

Posted in *nix, *nix-tools, postfix, Power User | Leave a Comment »

Viewing the last lines of the postfix log with journalctl (with help from Unix & Linux Stack Exchange)

Posted by jpluimers on 2022/02/24

Two command-lines I use to view my Postfix logs:

  1. journalctl --unit postfix --since "2 days ago"
  2. journalctl --unit postfix --pager-end

Note that neither of these work well with the --follow (or equivalent -f) option, as this will effectively disable the pager (which by default is less).

The second is via [Wayback] systemd – How to see the latest x lines from systemctl service log – Unix & Linux Stack Exchange (which got the--pagerend bit wrong, as it misses a dash and should be --pager-end, but still thanks [Wayback] Daniel Kmak):

Just:

journalctl -u SERVICE_NAME -e

Parameter -e stands for:

-e –pagerend

That’s the one ! Other answers will go through the whole log to get to its end, which can be veeeeery long for large syslogs.

The last bit (by [Wayback] Léo Germond, thanks!) is why I like it most.

Similarly, specifying --since in the first example will not go through the whole log.

Some background information:

Read the rest of this entry »

Posted in *nix, *nix-tools, bash, Development, journalctl and journald, Linux, postfix, Power User, Scripting, Software Development, systemd | Leave a Comment »

Some scripts and tips for easing the maintenance of a postfix based SMTP system

Posted by jpluimers on 2021/11/24

A few scripts and tips I found Googling around.

Deleting queued messages by regular expression pattern

I have seen the below delete-from-mailq.pl script numerous time, usually without any attribution (for instance [Wayback] Postfix Flush the Mail Queue – nixCraft and  [Wayback] postfix-delete.pl – Following script deletes all mail from the mailq which matches the regular expression specified as the first argument · GitHub).

The earliest version I could find was in [Wayback] ‘Re: delete messages from mailq’ – MARC by [Wayback] ‘Ralf Hildebrandt ‘ posts – MARC:

--- snip ---
#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)\*?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}
                                
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);
--- snip ---

And then use:
% delete-from-mailq "^test"

 

Tips

[Wayback] How do I check the postfix queue size? – Server Fault

Lots of great answers and pointers to useful guides/software there.

qstat

[Wayback] Postfix Bottleneck Analysis points to [Wayback] Postfix manual – qshape(1): qshape - Print Postfix queue domain and age distribution, then explains about different scenarion and queues:

postqueue

postqueue -p | tail -n 1

Last line in the postqueue -p shows how many requests and size:

-- 317788 Kbytes in 11860 Requests.

View queues size

I tried finding the original posting of the below script, but could not. If you find it, please let me know.

#!/usr/bin/env perl

# postfix queue/s size
# author: 
# source: http://tech.groups.yahoo.com/group/postfix-users/message/255133

use strict;
use warnings;
use Symbol;
sub count {
        my ($dir) = @_;
        my $dh = gensym();
        my $c = 0;
        opendir($dh, $dir) or die "$0: opendir: $dir: $!\n";
        while (my $f = readdir($dh)) {
                if ($f =~ m{^[A-F0-9]{5,}$}) {
                        ++$c;
                } elsif ($f =~ m{^[A-F0-9]$}) {
                        $c += count("$dir/$f");
                }
        }
        closedir($dh) or die "closedir: $dir: $!\n";
        return $c;
}
my $qdir = `postconf -h queue_directory`;
chomp($qdir);
chdir($qdir) or die "$0: chdir: $qdir: $!\n";
printf "Incoming: %d\n", count("incoming");
printf "Active: %d\n", count("active");
printf "Deferred: %d\n", count("deferred");
printf "Bounced: %d\n", count("bounce");
printf "Hold: %d\n", count("hold");
printf "Corrupt: %d\n", count("corrupt");

Various commands

[Wayback] Inspecting Postfix’s email queue – Tech-G explaining about:

  • mailq
  • postqueue -p
  • postcat -vq XXXXXXXXXX (where XXXXXXXXXX is the message ID)
  • postqueue -f / postfix flush
  • postsuper -d to delete messages

More of these in [Wayback] Postfix Mail Queue Management – Linux Hint and [Wayback] Postfix Bottleneck Analysis: queues.

Makefile

Based on [Wayback] Using “make” for Postfix file maintenance

MAPS = relays.db aliases.db transport.db relocated.db \
        virtual.db sender_checks.db rejected_recips.db \
        helo_access.db

all : $(MAPS)

aliases.db : aliases
        newaliases

%.db : %
        postmap $*

This is my Makefile that runs fine on Tumbleweed (note: all 8-space indents are TAB characters):

MAPS =  /etc/aliases.db \
        transport.db \
        virtual.db \
        helo_access.db \
        canonical.db \
        sasl_passwd.db \
        relocated.db \
        relay.db \
        access.db \
        relay_ccerts.db \
        sender_canonical.db

all : $(MAPS)

aliases.db : aliases
        @echo "Rebuilding $@."
        newaliases

%.db : %
        @echo "Rebuilding $@."
        postmap $*

In the future, I might try [Wayback] Makefile.postfix · GitHub, though I think it is convoluted:


## Postfix: Makefile to update *.db files
POSTCONF= /usr/sbin/postconf
POSTMAP= /usr/sbin/postmap
default: postmap
postmap: Makefile.postmap
@echo 'Updating database files …'
$(MAKE) -f Makefile.postmap
Makefile.postmap: main.cf
@echo 'Updating $@ …'
@set -e; \
rm -f $@.$$$$.tmp; \
echo 'POSTMAP=$(POSTMAP)' >>$@.$$$$.tmp; \
echo 'postmap::' >>$@.$$$$.tmp; \
config_directory="$(PWD)"; \
{ $(POSTCONF) -c $(PWD) || kill $$$$; } \
|tr ' ' '\n' \
|sed -n \
-e 's/,$$//' \
-e 's#^hash:\$$config_directory/##p' \
-e 's#^hash:'"$$config_directory/##p" \
|sort -u \
|while read mapfile; do \
echo "postmap:: $$mapfile.db" >>$@.$$$$.tmp; \
echo "$$mapfile.db: $$mapfile" >>$@.$$$$.tmp; \
echo " \$$(POSTMAP) $$<" >>$@.$$$$.tmp; \
done; \
mv $@.$$$$.tmp $@

 

 

[Wayback] Ralf Hildebrandt

Ralf Hildebrandt is an active and well-known figure in the Postfix community. He’s a systems engineer for T-NetPro, a German telecommunications company and has spoken about Postfix at industry conferences and contributes regularly to a number of open source mailing lists.

Co-author of this book: [Wayback: Book of Postfix State-of-the-Art Message Transport ISBN 9781593270018] (which used to have its own site: [Wayback: The Book of Postfix]

Book of Postfix

State-of-the-Art Message Transport

By Patrick KoetterRalf Hildebrandt

Publisher: No Starch PressRelease Date: March 2005Pages: 496

Best practices for Postfix–the popular alternative to Sendmail. Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail and comes preinstalled in many Linux distributions as the default mailer. The Book of Postfix is a complete guide to Postfix whether used at home, as a mailrelay or virus-scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from spam and viruses, managing multiple domains, and offering roaming access.

This is a great review of the book: [Wayback] The Book of Postfix (Ralf Hildebrandt, Patrick Koetter)

Related

For my postfix studies… « The Wiert Corner – irregular stream of stuff

–jeroen

 

Posted in *nix, *nix-tools, bash, Communications Development, Development, Internet protocol suite, Makefile, postfix, Power User, Scripting, SMTP, Software Development | Leave a Comment »

flush deferred messages in mail queue – SysAdmin Tips

Posted by jpluimers on 2021/03/19

[WayBack] flush deferred messages in mail queue – SysAdmin Tips

In Exim:

# exim -qff

In Postfix:

# postfix flush

In Sendmail:

# sendmail -OTimeout.hoststatus=0m -q -v

–jeroen

Posted in *nix, *nix-tools, exim mail, postfix, Power User, sendmail | Leave a Comment »

Inspecting Postfix’s email queue – Tech-G

Posted by jpluimers on 2021/03/08

[WayBack] Inspecting Postfix’s email queue – Tech-G:

mailq

or

postqueue -p

then

postcat -vq XXXXXXXXXX

Process (flush) the queue:

postqueue -f

OR

postfix flush

Viewing which senders/receivers are in the queue most:

Try with:
– qshape

More specific:
qshape -s deferred | head ===> deferred mails based on SENDER
qshape deferred | head ===> deferred mails based on RECEIVER

Rate limiting: [WayBack] Postfix Configuration Parameters: default_destination_rate_delay

–jeroen

Posted in *nix, *nix-tools, postfix, Power User | Leave a Comment »

 
%d bloggers like this: