Is there anything out there that does what md5sum does, only faster? I just want some kind of sum to be generated that can be used later to determine if the file is unchanged (not needing to create secure shadow passwords). Most files in my application will be very large (80+ GB) so speed is a concern.

The Fletcher checksums are a good compromise (as used in the ZFS filesystem, and the modified Adler version in zlib). A ones-complement addition is very fast, but not particularly robust.

However, the main bottleneck will be disk access, not the speed of the checksum computation.

You also have to weigh up the possibility of a checksum collision; if the file is different but the checksum is the same (which is more likely with shorter checksums), what will be the consequences in your application? There are good reasons for the popularity of MD5.

2 members found this post helpful.

I just tried md5sum vs Adler and md5sum is faster on a 1.5 GB file. I agree with neonsignal that what is taking so long is actually the disk I/O not the checksum.