Fastest Sha1 Function (Linux, up to 2GB files)

Question

Is there any linux command line implementation that performs exceptionally well for generating sha1's on large files (< 2GB)?

I have played around with 'openssl sha1' and it takes minutes to get the sha1 for a 2GB file : /.

Answer 1

On my machine, for a file of 1GB, with enough memory to have the entire file cached in memory after the first run:

sha1sum: 3.92s
openssl sha1: 3.48s
python hashlib.sha1: 3.22s

it takes minutes to get the sha1 for a 2GB file

There's something wrong there then, unless you're using incredibly slow old hardware. Even on the first run, where the file was being read directly from disc, it was only taking ‘openssl sha1’ about 20s per gig on my machine. Are you having slow I/O problems in general?

Answer 2

I don't think that a SHA algorithm could be optimized for size, since it operates on blocks of a fixed size, and the computation cannot be done in parallel. It seems that the fastest implementation on a small file will also be the fastest on a large file.