The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 1,497 other followers

Archive for the ‘*nix-tools’ Category

Follow up on “btrfs free space. It’s complicated. Still.”

Posted by jpluimers on 2019/02/14

In the mean time I’ve made a bit of progress on btrfs free space. It’s complicated. Still.

Let me start with an example system that has details further below.

  • total of quotas is slightly more than 1.1 Gibibyte
    • Sometimes this helps making the quota list better:
      btrfs quota rescan /
  • the disk partition itself is 10 Gibibyte
  • btrfs indicates there is 6.6 Gibibyte used
  • df indicates there is 11 Gigabyte total, 6.9 Gigabyte used and 2.6 Gigabyte available.

In short: the used 6.6 Gibibyte (which matches 6.9 Gigabyte) does not match the 11 Gibibyte. A situation very similar to [WayBackDisk usage is more than double the snapshots exclusive data — Linux BTRFS.

Reminder to self: try bedup that is supposed to deduplicate btrfs data: [WayBackRe: Disk usage is more than double the snapshots exclusive data — Linux BTRFS

I need to check out on de-duplication (as I know this particular machine has quite a bit of duplicate data).

But first lets get the size down a bit with this series of commands:

sftp-host:~ # btrfs balance start -dusage=0 -musage=0 /
Done, had to relocate 0 out of 18 chunks
sftp-host:~ # btrfs balance start -dusage=10 -musage=10 /
Done, had to relocate 1 out of 18 chunks
sftp-host:~ # btrfs balance start -dusage=20 -musage=20 /
Done, had to relocate 1 out of 18 chunks
sftp-host:~ # btrfs balance start -dusage=30 -musage=30 /
Done, had to relocate 2 out of 18 chunks
sftp-host:~ # btrfs balance start -dusage=40 -musage=40 /
Done, had to relocate 1 out of 17 chunks
sftp-host:~ # btrfs balance start -dusage=50 -musage=40 /
Done, had to relocate 2 out of 17 chunks
sftp-host:~ # btrfs balance start -dusage=60 -musage=40 /
Done, had to relocate 2 out of 17 chunks
sftp-host:~ # btrfs balance start -dusage=60 -musage=60 /
sftp-host:~ # btrfs filesystem show
Label: none  uuid: 6492a1c6-5fbc-4938-bf11-57d6194e6b8f
    Total devices 1 FS bytes used 6.61GiB
    devid    1 size 10.00GiB used 8.88GiB path /dev/sda2

sftp-host:~ # btrfs filesystem df /
Data, single: total=7.82GiB, used=6.35GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=512.00MiB, used=263.47MiB
GlobalReserve, single: total=22.67MiB, used=0.00B

Compare this to the initial situation:

sftp-host:~ # btrfs filesystem show
Label: none  uuid: 6492a1c6-5fbc-4938-bf11-57d6194e6b8f
    Total devices 1 FS bytes used 6.61GiB
    devid    1 size 10.00GiB used 10.00GiB path /dev/sda2

sftp-host:~ # btrfs filesystem df /
Data, single: total=8.94GiB, used=6.35GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=512.00MiB, used=264.27MiB
GlobalReserve, single: total=23.48MiB, used=0.00B

Now you see that:

  • far less of the partition is actually used by the filesystem (was 10 Gibibyte, now 8.88 Gibibyte)
  • far less storage is needed for the data (was 8.94 Gibibyte, now 7.82 Gibibyte to store 6.35 Gibibyte)

If the above succeeds

Continue with steps closer to 99 (which is a percentage) and if that succeeds try this:

btrfs balance start --full-balance /

In my experience it needs at least 60% free dh -f disk space to run to completion. If it fails, it’s no problem: it merges the final almost full blocks. But those blocks will be split soon anyway because of file system write activity.

Nicer overview

You can even get a nicer view by executing btrfs filesystem usage -T / (which I did after continuing up to 99):

Overall:
    Device size:          10.00GiB
    Device allocated:          8.22GiB
    Device unallocated:        1.78GiB
    Device missing:          0.00B
    Used:              7.00GiB
    Free (estimated):          2.72GiB  (min: 1.83GiB)
    Data ratio:               1.00
    Metadata ratio:           2.00
    Global reserve:       24.55MiB  (used: 48.00KiB)

             Data    Metadata  System              
Id Path      single  DUP       DUP      Unallocated
-- --------- ------- --------- -------- -----------
 1 /dev/sda2 7.41GiB 768.00MiB 64.00MiB     1.78GiB
-- --------- ------- --------- -------- -----------
   Total     7.41GiB 384.00MiB 32.00MiB     1.78GiB
   Used      6.47GiB 269.88MiB 16.00KiB

 

If the above fails

Three things to try now:

  1. Try to start with lower values of -dusage and -musage.
  2. Split-dusage and -musage in different btrfs balance start commands.
  3. Try to remove any snapper snapshots that you do not need.

Log:

sftp-host:~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        176M     0  176M   0% /dev
tmpfs           182M     0  182M   0% /dev/shm
tmpfs           182M  868K  181M   1% /run
tmpfs           182M     0  182M   0% /sys/fs/cgroup
/dev/sda2        11G  6.9G  2.6G  73% /
/dev/sda2        11G  6.9G  2.6G  73% /var/spool
/dev/sda2        11G  6.9G  2.6G  73% /tmp
/dev/sda2        11G  6.9G  2.6G  73% /boot/grub2/i386-pc
/dev/sda2        11G  6.9G  2.6G  73% /boot/grub2/x86_64-efi
/dev/sda2        11G  6.9G  2.6G  73% /var/crash
/dev/sda2        11G  6.9G  2.6G  73% /var/lib/named
/dev/sda2        11G  6.9G  2.6G  73% /var/opt
/dev/sda2        11G  6.9G  2.6G  73% /var/lib/mailman
/dev/sda2        11G  6.9G  2.6G  73% /var/tmp
/dev/sda2        11G  6.9G  2.6G  73% /var/log
/dev/sda2        11G  6.9G  2.6G  73% /var/lib/pgsql
/dev/sda2        11G  6.9G  2.6G  73% /var/lib/machines
/dev/sda2        11G  6.9G  2.6G  73% /srv
/dev/sda2        11G  6.9G  2.6G  73% /usr/local
/dev/sda2        11G  6.9G  2.6G  73% /opt
/dev/sda2        11G  6.9G  2.6G  73% /.snapshots
/dev/sda3       5.5G   36M  5.5G   1% /home
tmpfs            37M     0   37M   0% /run/user/1000
sftp-host:~ # btrfs filesystem show
Label: none  uuid: 6492a1c6-5fbc-4938-bf11-57d6194e6b8f
    Total devices 1 FS bytes used 6.61GiB
    devid    1 size 10.00GiB used 10.00GiB path /dev/sda2

sftp-host:~ # btrfs filesystem df /
Data, single: total=8.94GiB, used=6.35GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=512.00MiB, used=264.27MiB
GlobalReserve, single: total=23.48MiB, used=0.00B
sftp-host:~ # btrfs qgroup show /
qgroupid         rfer         excl 
--------         ----         ---- 
0/5          16.00KiB     16.00KiB 
0/257         1.05MiB      1.05MiB 
0/258         2.55GiB     51.11MiB 
0/259         2.36MiB      2.36MiB 
0/260        16.00KiB     16.00KiB 
0/261        16.00KiB     16.00KiB 
0/262        16.00KiB     16.00KiB 
0/263        36.00KiB     36.00KiB 
0/264        16.00KiB     16.00KiB 
0/265        16.00KiB     16.00KiB 
0/266        16.00KiB     16.00KiB 
0/267        16.00KiB     16.00KiB 
0/268        16.00KiB     16.00KiB 
0/269       533.83MiB    533.83MiB 
0/270        16.00KiB     16.00KiB 
0/271        48.00KiB     48.00KiB 
0/272        16.00KiB     16.00KiB 
0/289        16.00KiB     16.00KiB 
0/401         2.80GiB    396.91MiB 
0/402         2.55GiB      9.57MiB 
0/403         2.55GiB     12.91MiB 
0/404         2.54GiB    676.00KiB 
0/405         2.54GiB    660.00KiB 
0/406         2.81GiB     60.34MiB 
0/407         2.55GiB      8.66MiB 
0/408         2.55GiB      4.57MiB 
0/409         2.56GiB     24.31MiB 
0/410         2.55GiB      7.28MiB 
0/411         2.57GiB     20.55MiB 
255/289      16.00KiB     16.00KiB 
sftp-host:~ # !~
~/Versioned/btrfs-size/btrfs-size.sh 
===============================================================================================
Snapshot / Subvolume                                               ID   Total    Exclusive Data
===============================================================================================
257 gen 505741 top level 5 path .snapshots                         257  1.05MB   1.05MB   
258 gen 505796 top level 257 path .snapshots/1/snapshot            258  2.55GB   51.11MB  
259 gen 505736 top level 5 path boot/grub2/i386-pc                 259  2.36MB   2.36MB   
260 gen 452028 top level 5 path boot/grub2/x86_64-efi              260  16.00KB  16.00KB  
261 gen 452028 top level 5 path opt                                261  16.00KB  16.00KB  
262 gen 505720 top level 5 path srv                                262  16.00KB  16.00KB  
263 gen 505791 top level 5 path tmp                                263  36.00KB  36.00KB  
264 gen 505717 top level 5 path usr/local                          264  16.00KB  16.00KB  
265 gen 452028 top level 5 path var/crash                          265  16.00KB  16.00KB  
266 gen 452028 top level 5 path var/lib/mailman                    266  16.00KB  16.00KB  
267 gen 452028 top level 5 path var/lib/named                      267  16.00KB  16.00KB  
268 gen 452028 top level 5 path var/lib/pgsql                      268  16.00KB  16.00KB  
269 gen 505795 top level 5 path var/log                            269  533.83MB 533.83MB 
270 gen 452028 top level 5 path var/opt                            270  16.00KB  16.00KB  
271 gen 505796 top level 5 path var/spool                          271  48.00KB  48.00KB  
272 gen 505771 top level 5 path var/tmp                            272  16.00KB  16.00KB  
289 gen 452028 top level 5 path var/lib/machines                   289  16.00KB  16.00KB  
401 gen 451786 top level 257 path .snapshots/92/snapshot           401  2.81GB   396.91MB 
402 gen 465358 top level 257 path .snapshots/93/snapshot           402  2.55GB   9.57MB   
403 gen 465363 top level 257 path .snapshots/94/snapshot           403  2.55GB   12.91MB  
404 gen 471598 top level 257 path .snapshots/95/snapshot           404  2.54GB   676.00KB 
405 gen 471603 top level 257 path .snapshots/96/snapshot           405  2.54GB   660.00KB 
406 gen 471658 top level 257 path .snapshots/97/snapshot           406  2.81GB   60.34MB  
407 gen 487231 top level 257 path .snapshots/98/snapshot           407  2.55GB   8.66MB   
408 gen 490073 top level 257 path .snapshots/99/snapshot           408  2.55GB   4.57MB   
409 gen 490081 top level 257 path .snapshots/100/snapshot          409  2.56GB   24.31MB  
410 gen 505715 top level 257 path .snapshots/101/snapshot          410  2.55GB   7.28MB   
411 gen 505739 top level 257 path .snapshots/102/snapshot          411  2.57GB   20.55MB  
===============================================================================================
                                                                Exclusive Total: 1.11GB    
sftp-host:~ # 

–jeroen

Posted in *nix, *nix-tools, btrfs, File-Systems, Power User | Leave a Comment »

Ancient Unix based keyboard shortcuts

Posted by jpluimers on 2019/02/14

Somehow I’ve been doing Unix like work off and on for most of my life. So I know many ancient shell shortcuts (like Ctrl-E for end-of-line and Ctrl-A for start-of-line) by heart.

Many don’t, or even list shortcuts as if they are for the shell, but in fact aren’t.

So here are some links that distinguish between various kinds of keyboard shortcuts and give some insight in the history of these shortcuts (that go back into the 1970s, yes I’m that old).

–jeroen

via:

Posted in *nix, *nix-tools, Development, Power User, Software Development | Leave a Comment »

Some Postfix configuration guidelines

Posted by jpluimers on 2019/02/08

Not just for Postfix are the first two guidelines:

  • Change one thing at a time
  • Save known working configurations

For the latter, I’m using etckeeper pushing to an external git repository hoster.

For Postfix are the others from [WayBackPostfix Configuration Guidelines.

One tip that’s missing, but saved my life numerous of times:

In /etc/postfix/main.cfg do not use this line ever:

inet_interfaces = $myhostname

If the resolving (through DNS or hosts file) of $myhostname fails for any reason in the future, then Postfix will not start at all, but in stead emit a fatal error like this:

/usr/sbin/postconf: fatal: parameter inet_interfaces: no local interface found for 127.0.0.2

Specify exact interfaces in stead, like any of these:

inet_interfaces = all

inet_interfaces = localhost

inet_interfaces = 192.168.24.68

–jeroen

Posted in *nix, *nix-tools, postfix, Power User | Leave a Comment »

‪Dear #lazyweb, can anyone point me to a modern email server setup (just emai…

Posted by jpluimers on 2019/02/01

Summary from [WayBack]‪ Dear #lazyweb, can anyone point me to a modern email server setup (just email) with letsencrypt, some spam filter, multi domain preferably on RHEL/Cent… – Jan Wildeboer – Google+

  • many SMTP servers on the interwebs do not have proper TLS setups, so do not require remote SMTP servers to deliver email with a proper certificate
  • delivering mail via SMTP using STARTTLS with a proper certificate yourself is a good step forward
  • postfix
  • dovecot
  • greylisting (although in practice it does not make much of a difference any more)
  • fail2ban
  • dnsbl (often called rbl)
  • spamassasin
  • rspamd (supports SPF, DKIM and many others)
  • letsencrypt automation can be tough, so here is a small wrapper: [WayBack] GitHub – DrGlitchMX/update-letsencrypt: Tiny script for updating “Let’s Encrypt!” certificates from cron
  • it helps having letsencrypt and the mail server to be on one machine:
    • multidomain let’s encrypt cert that has my webserver name and the mailserver in the Subject Alternative Names field. As both are on the same machine certbot can automatically update it and I just point Postfix and Dovecot to the LE files.
  • Hans-Martin Mosner SMTP as-is is just not suitable for the kind of decentralized mail that you would prefer. You need some mechanism to determine which mail senders to trust and which not. Cryptography is suitable at the MUA level and should be used much more, but at the MTA level, TLS for privacy and SPF(bleh) or DKIM(meh) for sender domain authentication are basically your only weapons -much too weak. The PGP web of trust must be considered a failed experiment – who of your mail contacts uses PGP properly or at all? Ironically the only secure messaging solutions for the masses are centralized.

Things to do:

  • find a proper multi-MX fallback setup guide for postfix

–jeroen

Read the rest of this entry »

Posted in *nix, *nix-tools, Communications Development, Development, Internet protocol suite, postfix, Power User, SMTP | Leave a Comment »

linux – Test if a port on a remote system is reachable (without telnet) – Super User

Posted by jpluimers on 2019/01/29

Just learned that bash can do TCP and UDP itself:

Bash has been able to access TCP and UDP ports for a while. From the man page:

/dev/tcp/host/port
    If host is a valid hostname or Internet address, and port is an integer port number
    or service name, bash attempts to open a TCP connection to the corresponding socket.
/dev/udp/host/port
    If host is a valid hostname or Internet address, and port is an integer port number
    or service name, bash attempts to open a UDP connection to the corresponding socket.

So you could use something like this:

xenon-lornix:~> cat < /dev/tcp/127.0.0.1/22
SSH-2.0-OpenSSH_6.2p2 Debian-6
^C pressed here

Taa Daa!

This for systems that do not have telnet installed (Windows stopped using this a long time ago, many Linux distributions followed suit) and you cannot to use nc (also known as netcat).

–jeroen: [WayBacklinux – Test if a port on a remote system is reachable (without telnet) – Super User

Posted in *nix, *nix-tools, bash, bash, Development, Power User, Scripting, Software Development | Leave a Comment »

 
%d bloggers like this: