Parallel hashing on MacOS
Posted by jpluimers on 2025/04/14
Now that I have had an Apple silicon for a while, which has enough cores to perform parallel work, this is how I calculated a bunch of hashed from a lot of large files:
find . -type f | xargs -P 0 -n 1 md5 -rfind . -type f | xargs -P 0 -n 1 shasum --algorithm 1find . -type f | xargs -P 0 -n 1 shasum --algorithm 256
I contemplated about using GNU parallel, but that is not installed by default on MacOS and I was already familiar with xargs.
Argument meanings can be found at these locations:
Related:
- [Wayback/Archive] hashsum – How can I parallelize command sha256sum or other hashing commands? – Unix & Linux Stack Exchange (thanks [Wayback/Archive] Berkant E., [Wayback/Archive] Ole Tange – parallel author – and [Wayback/Archive] cas)
Q
I want to parallelize hash calculation process because I have a very large amount of file counts and sizes. When I see CPU usage of these commands, I get upset because they are only using one thread; how can I parallelize these?
sha256sum foo.mp4 OR openssl -dgst sha256 foo.mp4A
For parallelizing across files you can use GNU Parallel:
parallel sha256sum ::: *Parallelizing hashing for a single file can be done with certain hash functions using a Merkle tree.
b3sumis such a tool.A
xargshas a-Poption for running multiple jobs in parallel. It’s nowhere near as flexible as Ole Tange’sparallelprogram, but it’s fine for most simple parallelisation tasks.For example:
find . -name '*.mp4' -print0 | xargs -0r -n 1 -P 0 openssl dgst -sha256…
Note:
-Pis not a standard POSIX option for xargs. Requires GNU or *BSD xargs. Maybe some other versions too.The last answer also contains some hints on how to calculate
-Pand-nvalues. - find:
- xargs:
- parallel:
- [Wayback/Archive] GNU Parallel – Summary [Savannah]
- [Wayback/Archive] GNU Parallel Tutorial — GNU Parallel 20241022 documentation
- [Wayback/Archive] GNU Parallel 2018 book
- [Wayback/Archive] Part 1: GNU Parallel script processing and execution – YouTube
- [Wayback/Archive] parallel — Homebrew Formulae
brew install parallel
- [Wayback/Archive] md5 Man Page – macOS – SS64.com
- [Wayback/Archive] shasum: Print or Check SHA Checksums | perl-Digest-SHA Commands | Man Pages | ManKier
Queries:
- [Wayback/Archive] linux parallel hash – Sök på Google
- [Wayback/Archive] macos parallel hash – Google Suche
- [Wayback/Archive] brew install parallel – Google Suche
- [Wayback/Archive] macos man page md5 – Sök på Google
--jeroen
The Wiert Corner - irregular stream of stuff 
Leave a comment