The Wiert Corner – irregular stream of stuff

Now that I have had an Apple silicon for a while, which has enough cores to perform parallel work, this is how I calculated a bunch of hashed from a lot of large files:

• find . -type f | xargs -P 0 -n 1 md5 -r
• find . -type f | xargs -P 0 -n 1 shasum --algorithm 1
• find . -type f | xargs -P 0 -n 1 shasum --algorithm 256

I contemplated about using GNU parallel, but that is not installed by default on MacOS and I was already familiar with xargs.

Argument meanings can be found at these locations:

• find -type
• xargs -P
• xargs -n
• md5 -r
• shasum --algorithm

Related:

• [Wayback/Archive] hashsum – How can I parallelize command sha256sum or other hashing commands? – Unix & Linux Stack Exchange (thanks [Wayback/Archive] Berkant E., [Wayback/Archive] Ole Tange – parallel author – and [Wayback/Archive] cas)
  

  Q

  I want to parallelize hash calculation process because I have a very large amount of file counts and sizes. When I see CPU usage of these commands, I get upset because they are only using one thread; how can I parallelize these?

  sha256sum foo.mp4
  OR
  openssl -dgst sha256 foo.mp4

  A

  For parallelizing across files you can use GNU Parallel:

  parallel sha256sum ::: *
  

  Parallelizing hashing for a single file can be done with certain hash functions using a Merkle tree. b3sum is such a tool.

  A

  xargs has a -P option for running multiple jobs in parallel. It’s nowhere near as flexible as Ole Tange’s parallel program, but it’s fine for most simple parallelisation tasks.

  For example:

  find . -name '*.mp4' -print0 | xargs -0r -n 1 -P 0 openssl dgst -sha256

  …

  Note: -P is not a standard POSIX option for xargs. Requires GNU or *BSD xargs. Maybe some other versions too.

  The last answer also contains some hints on how to calculate -P and -n values.

•  find:
  • [Wayback/Archive] find: find files | POSIX Commands | Man Pages | ManKier
  • [Wayback/Archive] find: search for files in a directory hierarchy | findutils Commands | Man Pages | ManKier
• xargs:
  • [Wayback/Archive] xargs: construct argument lists and invoke utility | POSIX Commands | Man Pages | ManKier
  • [Wayback/Archive] xargs: build and execute command lines from standard input | findutils Commands | Man Pages | ManKier
• parallel:
  • [Wayback/Archive] GNU Parallel – Summary [Savannah]
  • [Wayback/Archive] GNU Parallel Tutorial — GNU Parallel 20241022 documentation
  • [Wayback/Archive] GNU Parallel 2018 book
    • [Wayback/Archive] GNU_Parallel_2018.pdf
  • [Wayback/Archive] Part 1: GNU Parallel script processing and execution – YouTube
  • [Wayback/Archive] parallel — Homebrew Formulae
    

    brew install parallel

• [Wayback/Archive] md5 Man Page – macOS – SS64.com
• [Wayback/Archive] shasum: Print or Check SHA Checksums | perl-Digest-SHA Commands | Man Pages | ManKier

Queries:

• [Wayback/Archive] linux parallel hash – Sök på Google
• [Wayback/Archive] macos parallel hash – Google Suche
• [Wayback/Archive] brew install parallel – Google Suche
• [Wayback/Archive] macos man page md5 – Sök på Google

--jeroen