.net File.Copy very slow when copying many small files (not over network)

Question

I'm making a simple folder sync backup tool for myself and ran into quite a roadblock using File.Copy. Doing tests copying a folder of ~44,000 small files (Windows mail folders) to another drive in my system, I found that using File.Copy was over 3x slower than using a command line and running xcopy to copy the same files/folders. My C# version takes over 16+ minutes to copy the files, whereas xcopy takes only 5 minutes. I've tried searching for help on this topic, but all I find is people complaining about slow file copying of large files over a network. This is neither a large file problem nor a network copying problem.

I found an interesting article about a better File.Copy replacement, but the code as posted has some errors which causes problems with the stack and I am nowhere near knowledgeable enough to fix the problems in his code.

Are there any common or easy ways to replace File.Copy with something more speedy?

Answer 1

There are two algorithms for faster file copy:

If source and destination are different disks Then:

• One thread reading files continuously and storing in a buffer.
• Another thread writing files continuously from that buffer.

If source and destination is same disk then:

• Read a fixed chunk of bytes, say 8K at a time, no matter how many files that is.
• Write that fixed chunk to destination, either in one file or in multiple files.

This way you will get significant performance.

Alternative is you just invoke xcopy from your .net code. Why bother doing it using File.Copy. You can capture xcopy output using Process.StandardOutput and show on the screen in order to show user what's going on.

Answer 2

One thing to consider is whether your copy has a user interface that updates during the copy. If so, make sure your copy is running on a separate thread, or both your UI will freeze up during the copy, and the copy will be slowed down by making blocking calls to update the UI.

I have written a similar program and in my experience, my code ran faster than a windows explorer copy (not sure about xcopy from the command prompt).

Also if you have a UI, don't update on every file; instead update every X megabytes or every Y files (whichever comes first), this keeps down the amount of updating to something the UI can actually handle. I used every .5MB or 10 files; those may not be optimal but it noticeably increased my copy speed and UI responsiveness.

Another way to speed things up is to use the Enumerate functions instead of Get functions (e.g. EnumerateFiles instead of GetFiles). These functions start returning results as soon as possible instead of waiting to return everything when the list is finished being built. They return an Enumerable, so you can just call foreach on the result: foreach(string file in System.IO.Directory.EnumerateDirectories(path)). For my program this also made a noticeable difference in speed, and would be even more helpful in cases like yours where you are dealing with directories containing many files.