Menu
I am writing a web crawler to search for files and download. My problem is I do not want to download the same files that are downloaded already to the local drive. I know it's possible to use the MD5 hash to compare but how can I do this on HTTP URL without downloading them to the local disk?
If this approach is wrong. Please advice on a better solution
485
Type the following command: md5sum [type file name with extension here] [path of the file] -- NOTE: You can also drag the file to the terminal window instead of typing the full path. Hit the Enter key. You'll see the MD5 sum of the file. Match it against the original value.
In Linux, the md5sum program computes and checks MD5 hash values of a file. It is a constituent of GNU Core Utilities package, therefore comes pre-installed on most, if not all Linux distributions.
Generally, two files can have the same md5 hash only if their contents are exactly the same. Even a single bit of variation will generate a completely different hash value. There is one caveat, though: An md5 sum is 128 bits (16 bytes).
Unless the webserver has some sort of service that shares the MD5, then No.
Computing a file hash requires every byte in the file. This is why changing a single byte changes the hash, to prevent getting modified files.
102
To generate a hash you're going to need the data (ie, you'll need to download it somehow).
I would suggest that you investigate using the If-Modified-Since HTTP header instead (or maybe ETag/If-None-Match, if the particular server provides it).
32
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
