Removing duplicate files with Powershell

Question

I have several thousand duplicate files (jar files as an example) that I'd like to use powershell to

1. Search through the file system recursively
2. Find the dups (either by name only or a checksum method or both)
3. Delete all duplicates but one.

I'm new to powershell and am throwing this out there to the PS folks that might be able to help.

Answer 1

try this:

ls *.txt -recurse | get-filehash | group -property hash | where { $_.count -gt 1 } | % { $_.group | select -skip 1 } | del

from: http://n3wjack.net/2015/04/06/find-and-delete-duplicate-files-with-just-powershell/

Answer 2

Keep a dictionary of files, delete when the next file name was already encountered before:

$dict = @{};
dir c:\admin -Recurse | foreach {
  $key = $_.Name #replace this with your checksum function
  $find = $dict[$key];
  if($find -ne $null) {
    #current file is a duplicate
    #Remove-Item -Path $_.FullName ?    
  }
  $dict[$key] = 0; #dummy placeholder to save memory
}

I used file name as a key, but you can use a checksum if you want (or both) - see code comment.