Can I prevent duplicate content using md5?

Question

I would like to prevent duplicate content. I do not want to keep a copies of content, so I decided to keep just the md5 signatures.

I read that md5 collisions do happen, different content could give in the same md5 signature.

Do you think md5 is enough?

Should I use md5 and sh1 together?

Answer 1

People have been able to deliberately produce MD5 collisions under contrived circumstances, but for preventing duplicate content (in the absence of malicious users) it's more than adequate.

Having said that, if you can use SHA-1 (or SHA-2) you should - you'll be fractionally but measurably safer from collisions.

Answer 2

MD5 should be fine, collisions are very rare, but if you're really worried, you can use sha-1 as well.

Though I guess the signatures really aren't that large, so if you have the spare processing cycles and the disk space, you could do both. But if space or speed is limited, I'd just go with one.