Bash command to (re)move duplicate files [eg photos]

Question

If I have a directory with a bunch of photos and some of them are duplicates [in everything except name], is there a way I can get a list of uniques and move them to another dir?

Eg

find . -type f -print0 | xargs -0 md5sum

that will give me a list of "md5 filename"

Now I just want to look at uniques based on that... eg pipe that to sort -u.

After that I want to mv all of those files somewhere else, but I can worry about that later...

Answer 1

You can use fdupes:

fdupes -r .

to get a list of duplicates. The move should be possible with some command chaining.

fdupes -r -f .

Shows you only the duplicated files. So if you have an image twice. You'll get one entry instead of both duplicated paths.

To move you could do:

for file in $(fdupes -r -f . | grep -v '^$')
do
  mv "$file" duplicated-files/
done

But be aware of name clashes..

Answer 2

From there:

sort | uniq -w 32

Will compare only the first 32 characters, which I believe should be the md5sum itself.