The Wiert Corner – irregular stream of stuff

Jeroen W. Pluimers on .NET, C#, Delphi, databases, and personal interests

  • My badges

  • Twitter Updates

  • My Flickr Stream

  • Pages

  • All categories

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 2,422 other followers

Verifying large sets of file hashes with md5sum

Posted by jpluimers on 2020/07/24

A few tips:

  1. Recursively getting all md5 sums from a source directory:

    cd /sourceDirectory
    find -type f \( -not -name "md5sum.txt" \) -exec md5sum '{}' \; > md5sum.txt


  2. Checking the sums against a target directory

    cd /targetDirectory
    md5sum -c /sourceDirectory/md5sum.txt


On some systems (this was an ESXi system which can’t run stuff from the console in parallel), you could optimise this using xargs for the generation and GNU parallel for the generation and checking. Both should be very similar:

GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: