Comparison between two big directories

Question

I have a large directory that contains only stuff in CS and Math. It is over 16GB in size. The types are text, png, pdf and chm. I have currently two branches: a branch of my brother's and mine. The initial files were the same. I need to compare them. I have tried to use Git, but there is a long loading time.

What is the best way to compare two big directories?

[Mixed Solution]

1. Do a "ls -R > different_files" in both directories [1]
2. "sdiff <(echo file1 | md5deep) <(echo file2 | md5deep)" [2]

What do you think? Any drawbacks?

[1] thanks to Paul Tomblin [2] great thanks to all repliers!

Answer 1

Use fslint: website. One of the options of the tool is "Duplicates". As per the description from the site: One of the most commonly used features of FSlint is the ability to find duplicate files. The easiest way to remove lint from a hard drive is to discard any duplicate files that may exist. Often a computer user may not know that they have four, five, or more copies of the exact same song in their music collection under different names or directories. Any file type whether it be music, photos, or work documents can easily be copied and replicated on your computer. As the duplicates are collected, they eat away at the available hard drive space. The first menu option offered by FSlint allows you to find and remove these duplicate files.

Answer 2

How to compare 2 folders without pre-existing commands/products:

Simply create a program that scans each directory and creates a file hash of each file. It outputs a file with each relative file path and the file hash.

Run this program on both folders.

Then you simply compare the 2 output files to see if they are the same. To compare those 2 files you just load them into a string and do a string compare.

The hashing algorithm you use doesn't matter. You can use MD5, SHA, CRC, ... You could also use the file size in the output files to help reduce the chance of collisions.

How to compare 2 folders with pre-existing commands/products:

Now if you just want a program that does it, use diff -r or windiff for windows based systems.