Verifying large sets of file hashes with md5sum
Posted by jpluimers on 2020/07/24
A few tips:
- Recursively getting all md5 sums from a source directory:
cd /sourceDirectory
find -type f \( -not -name "md5sum.txt" \) -exec md5sum '{}' \; > md5sum.txt.
- Checking the sums against a target directory
cd /targetDirectory
md5sum -c /sourceDirectory/md5sum.txt.
On some systems (this was an ESXi system which can’t run stuff from the console in parallel), you could optimise this using xargs for the generation and GNU parallel for the generation and checking. Both should be very similar:
GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel.
Via:
- [WayBack] command line – How can I recursively list Md5sum of all the files in a directory and its subdirectories? – Ask Ubuntu
- [WayBack] GNU Parallel – GNU Project – Free Software Foundation
- [WayBack] bash – parallel check md5 file – Stack Overflow
- [WayBack] Bash: parallelize md5sum checksum on many files – Stack Overflow
–jeroen
Leave a Comment