Efficient way to fingerprint an image (jpg, png, etc)?

Question

Is there an efficient way to get a fingerprint of an image for duplicate detection?

That is, given an image file, say a jpg or png, I'd like to be able to quickly calculate a value that identifies the image content and is fairly resilient to other aspects of the image (eg. the image metadata) changing. If it deals with resizing that's even better.

[Update] Regarding the meta-data in jpg files, does anyone know if it's stored in a specific part of the file? I'm looking for an easy way to ignore it - eg. can I skip the first x bytes of the file or take x bytes from the end of the file to ensure I'm not getting meta-data?

Answer 1

Stab in the dark, if you are looking to circumvent meta-data and size related things:

1. Edge Detection and scale-independent comparison
2. Sampling and statistical analysis of grayscale/RGB values (average lum, averaged color map)
3. FFT and other transforms (Good article Classification of Fingerprints using FFT)

And numerous others.

Basically:

1. Convert JPG/PNG/GIF whatever into an RGB byte array which is independent of encoding
2. Use a fuzzy pattern classification method to generate a 'hash of the pattern' in the image ... not a hash of the RGB array as some suggest
3. Then you want a distributed method of fast hash comparison based on matching threshold on the encapsulated hash or encoding of the pattern. Erlang would be good for this :)

Advantages are:

1. Will, if you use any AI/Training, spot duplicates regardless of encoding, size, aspect, hue and lum modification, dynamic range/subsampling differences and in some cases perspective

Disadvantages:

1. Can be hard to code .. something like OpenCV might help
2. Probabilistic ... false positives are likely but can be reduced with neural networks and other AI
3. Slow unless you can encapsulate pattern qualities and distribute the search (MapReduce style)

Checkout image analysis books such as:

1. Pattern Classification 2ed
2. Image Processing Fundamentals
3. Image Processing - Principles and Applications

And others

If you are scaling the image, then things are simpler. If not, then you have to contend with the fact that scaling is lossy in more ways than sample reduction.

Answer 2

Using the byte size of the image for comparison would be suitable for many applications. Another way would be to:

1. Strip out the metadata.
2. Calculate the MD5 (or other suitable hashing algorithm) for the image.
3. Compare that to the MD5 (or whatever) of the potential dupe image (provided you've stripped out the metadata for that one too)