How do I force Robocopy to overwrite files?

Question

In general, Robocopy ignores files for which lastwrittendate and filesize are the same. How can we escape this design? I'd like to force overwriting with Robocopy.

I expected that dst\sample.txt should be written test001. But these file are recognized as the same files by Robocopy and not overwritten. The "/IS" option is not effective in this case.

New-Item src -itemType Directory New-Item dst -itemType Directory New-Item src\sample.txt -itemType File -Value "test001" New-Item dst\sample.txt -itemType File -Value "test002" Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00" Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00"  ROBOCOPY.exe src dst /COPYALL /MIR Get-Content src\sample.txt, dst\sample.txt > test001 > test002  ROBOCOPY.exe src dst /COPYALL /MIR /IS Get-Content src\sample.txt, dst\sample.txt > test001 > test002 

Answer 1

From the documentation:

/is Includes the same files. /it Includes "tweaked" files.

"Same files" means files that are identical (name, size, times, attributes). "Tweaked files" means files that have the same name, size, and times, but different attributes.

robocopy src dst sample.txt /is      # copy if attributes are equal robocopy src dst sample.txt /it      # copy if attributes differ robocopy src dst sample.txt /is /it  # copy irrespective of attributes 

This answer on Super User has a good explanation of what kind of files the selection parameters match.

With that said, I could reproduce the behavior you describe, but from my understanding of the documentation and the output robocopy generated in my tests I would consider this a bug.

PS C:\temp> New-Item src -Type Directory >$null PS C:\temp> New-Item dst -Type Directory >$null PS C:\temp> New-Item src\sample.txt -Type File -Value "test001" >$null PS C:\temp> New-Item dst\sample.txt -Type File -Value "test002" >$null PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00" PS C:\temp> Set-ItemProperty dst\sample.txt -Name LastWriteTime -Value "2016/1/1 15:00:00" PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir ...   Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30  ------------------------------------------------------------------------------                             1    C:\temp\src\             Modified                   7        sample.txt  ------------------------------------------------------------------------------                 Total    Copied   Skipped  Mismatch    FAILED    Extras     Dirs :         1         0         0         0         0         0    Files :         1         1         0         0         0         0    Bytes :         7         7         0         0         0         0 ... PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir ...   Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30  ------------------------------------------------------------------------------                             1    C:\temp\src\             Same                       7        sample.txt  ------------------------------------------------------------------------------                 Total    Copied   Skipped  Mismatch    FAILED    Extras     Dirs :         1         0         0         0         0         0    Files :         1         1         0         0         0         0    Bytes :         7         7         0         0         0         0 ... PS C:\temp> Get-Content .\src\sample.txt test001 PS C:\temp> Get-Content .\dst\sample.txt test002

The file is listed as copied, and since it becomes a same file after the first robocopy run at least the times are synced. However, even though seven bytes have been copied according to the output no data was actually written to the destination file in both cases despite the data flag being set (via /copyall). The behavior also doesn't change if the data flag is set explicitly (/copy:d).

I had to modify the last write time to get robocopy to actually synchronize the data.

PS C:\temp> Set-ItemProperty src\sample.txt -Name LastWriteTime -Value (Get-Date) PS C:\temp> robocopy src dst sample.txt /is /it /copyall /mir ...   Options : /S /E /COPYALL /PURGE /MIR /IS /IT /R:1000000 /W:30  ------------------------------------------------------------------------------                             1    C:\temp\src\ 100%        Newer                      7        sample.txt  ------------------------------------------------------------------------------                 Total    Copied   Skipped  Mismatch    FAILED    Extras     Dirs :         1         0         0         0         0         0    Files :         1         1         0         0         0         0    Bytes :         7         7         0         0         0         0 ... PS C:\temp> Get-Content .\dst\sample.txt test001

An admittedly ugly workaround would be to change the last write time of same/tweaked files to force robocopy to copy the data:

& robocopy src dst /is /it /l /ndl /njh /njs /ns /nc |   Where-Object { $_.Trim() } |   ForEach-Object {     $f = Get-Item $_     $f.LastWriteTime = $f.LastWriteTime.AddSeconds(1)   } & robocopy src dst /copyall /mir 

Switching to xcopy is probably your best option:

& xcopy src dst /k/r/e/i/s/c/h/f/o/x/y 

Answer 2

This is really weird, why nobody is mentioning the /IM switch ?! I've been using it for a long time in backup jobs. But I tried googling just now and I couldn't land on a single web page that says anything about it even on MS website !!! Also found so many user posts complaining about the same issue!!

Anyway.. to use Robocopy to overwrite EVERYTHING what ever size or time in source or distination you must include these three switches in your command (/IS /IT /IM)

/IS :: Include Same files. (Includes same size files) /IT :: Include Tweaked files. (Includes same files with different Attributes) /IM :: Include Modified files (Includes same files with different times). 

This is the exact command I use to transfer few TeraBytes of mostly 1GB+ files (ISOs - Disk Images - 4K Videos):

robocopy B:\Source D:\Destination /E /J /COPYALL /MT:1 /DCOPY:DATE /IS /IT /IM /X /V /NP /LOG:A:\ROBOCOPY.LOG 

I did a small test for you .. and here is the result:

               Total    Copied   Skipped  Mismatch    FAILED    Extras     Dirs :      1028      1028         0         0         0       169    Files :      8053      8053         0         0         0         1    Bytes : 649.666 g 649.666 g         0         0         0   1.707 g    Times :   2:46:53   0:41:43                       0:00:00   0:41:44      Speed :           278653398 Bytes/sec.    Speed :           15944.675 MegaBytes/min.    Ended : Friday, August 21, 2020 7:34:33 AM 

Dest, Disk: WD Gold 6TB (Compare the write speed with my result)

Even with those "Extras", that's for reporting only because of the "/X" switch. As you can see nothing was Skipped and Total number and size of all files are equal to the Copied. Sometimes It will show small number of skipped files when I abuse it and cancel it multiple times during operation but even with that the values in the first 2 columns are always Equal. I also confirmed that once before by running a PowerShell script that scans all files in destination and generate a report of all time-stamps.

Some performance tips from my history with it and so many tests & troubles!:

. Despite of what most users online advise to use maximum threads "/MT:128" like it's a general trick to get the best performance ... PLEASE DON'T USE "/MT:128" WITH VERY LARGE FILES ... that's a big mistake and it will decrease your drive performance dramatically after several runs .. it will create very high fragmentation or even cause the files system to fail in some cases and you end up spending valuable time trying to recover a RAW partition and all that nonsense. And above all that, It will perform 4-6 times slower!!

For very large files:

1. Use Only "One" thread "/MT:1" | Impact: BIG
2. Must use "/J" to disable buffering. | Impact: High
3. Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: Medium.
4. Put the "/LOG:file" on a separate drive from the source or destination | Impact: Low.

For regular big files:

1. Use multi threads, I would not exceed "/MT:4" | Impact: BIG
2. IF destination disk has low Cache specs use "/J" to disable buffering | Impact: High
3. & 4 same as above.

For thousands of tiny files:

1. Go nuts :) with Multi threads, at first I would start with 16 and multibly by 2 while monitoring the disk performance. Once it starts dropping I'll fall back to the prevouse value and stik with it | Impact: BIG
2. Don't use "/J" | Impact: High
3. Use "/NP" with "/LOG:file" and Don't output to the console by "/TEE" | Impact: HIGH.
4. Put the "/LOG:file" on a separate drive from the source or destination | Impact: HIGH.