Optimizing Dictionary.EnumerateFiles over network

Question

I currently have a program that scans network shares. In order to do so, it first enumerates all the files and directories on the share. This is a very slow process. I currently use the below code, taken from a 2011 answer on this site.

static class SafeWalk
    {
        public static IEnumerable<string> EnumerateFiles(string path, string searchPattern, SearchOption searchOpt)
        {
            try
            {
                var dirFiles = Enumerable.Empty<string>();
                if (searchOpt == SearchOption.AllDirectories)
                {
                    dirFiles = Directory.EnumerateDirectories(path)
                                        .SelectMany(x => EnumerateFiles(x, searchPattern, searchOpt));
                }
                return dirFiles.Concat(Directory.EnumerateFiles(path, searchPattern));
            }
            catch (UnauthorizedAccessException ex)
            {
                Console.WriteLine(ex.Message);
                return Enumerable.Empty<string>();
            }
        }
    }

The problem is that everything else in the program is multi-threaded and optimized for speed. This is the only area that is seriously holding me back. It can takes a score of minutes to enumerate the files in the network share. This is on an intranet, and there are gigabit connections or greater between my machine and the server.

I did speed it up a good bit when I mapped the network path to a drive temporarily at runtime. Is there anything I can do to make this go faster? Looking at the resource monitor, it is barely using any CPU, memory or network bandwidth.

Answer 1

Consider using PInvoke to call FindFirstFileEx with the option FIND_FIRST_EX_LARGE_FETCH. According to Raymond Chen this flag is made for your situation.