Comparing images to find duplicates

Question

I have a few (38000) picture/video files in a folder. Approximately 40% of these are duplicates which I'm trying to get rid of. My question is, how can I tell if 2 files are identical? So far I tried to use a SHA1 of the files but it turns out that many duplicates files had different hashes. This is the code I was using:

public static String getHash(File doc) {
    MessageDigest md = null;
    try {
        md = MessageDigest.getInstance("SHA1");
        FileInputStream inStream = new FileInputStream(doc);
        DigestInputStream dis = new DigestInputStream(inStream, md);
        BufferedInputStream bis = new BufferedInputStream(dis);
        while (true) {
            int b = bis.read();
            if (b == -1)
                break;
        }

        inStream.close();
        dis.close();
        bis.close();
    } catch (NoSuchAlgorithmException | IOException e) {
        e.printStackTrace();
    }

    BigInteger bi = new BigInteger(md.digest());

    return bi.toString(16);
}

Can I modify this in any way? Or will I have to use a different method?

Answer 1

As outlined above duplicate detection can be based on a hash. However, if you want to have near duplicate detection, which means that you are searching for images that basically show the same things, but have been scaled, rotated, etc. you might need a content based image retrieval approach. There's LIRE (https://code.google.com/p/lire/), a Java library for that, and you'll find the "SimpleApplication" in the Download section. What you then can do is to

1. Index the first image
2. go to the next image I
3. Search for I in the index
4. If there are results with a score below a threshold, then mark them as duplicate
5. Index I
6. Go to (2)

Students of mine did it, it worked well, but I don't have the source code at hand. But rest assured, it's just a few lines and the simple application will get you started.