DirectoryInfo.GetFiles slow when using SearchOption.AllDirectories

Question

I am searching a moderate number (~500) of folders for a large number (~200,000) of files from a .NET application.

I hoped to use DirectoryInfo.GetFiles, passing in SearchOption.AllDirectories. However this approach seems to be a lot slower than writing my own code to iterate through the directories and do GetFiles just passing in a searchPattern.

Related MSDN info:

• GetFiles(String)
  Returns a file list from the current directory matching the given searchPattern.
• GetFiles(String, SearchOption)
  Returns a file list from the current directory matching the given searchPattern and using a value to determine whether to search subdirectories.

Has anyone had a similar experience to this?

Answer 1

These two functions are actually infamous for their performance. The reason is that GetFiles walks entire directory tree and constructs an array of FileInfo objects, and only then returns the result to the caller. Construction of said array involves a lot of memory allocations (I'm sure they use List internally, but still) since the number of entries cannot be known ahead of time.

If you're really into performance, you can P/Invoke into FindFirstFile/FindNextFile/FindClose, abstract them into an IEnumerable<FileInfo> and yield FileInfos one at a time.

Answer 2

The approach Anton mentioned using FindFirstFile() and related native methods has been implemented as of .NET 4 via DirectoryInfo.EnumerateFiles() so no more need for P/Invoke for this!