Parallel Loops for Folder Size Determination

Question

The overall programs goal is to determine the size of main folders in directories. It works pretty well for small drives, but struggles for larger ones. It took over 3 hours for one of the drives that I absolutely need. This is a copy of my folder sizing program, I am using.

    public  double getDirectorySize(string p)
    {

        //get array of all file names
        string[] a = Directory.GetFiles(p, "*.*", SearchOption.AllDirectories);

        //calculate total bytes in loop
        double b = 0;
        foreach (string name in a)
        {

            if (name.Length < 250) // prevents path too long errors
            {


                    //use file info to get length of each file 
                    FileInfo info = new FileInfo(name);
                    b += info.Length;
            }
        }

        //return total size
        return b;
    }

So what I was thinking of using parallel loops in a form of parallel foreach loops. Each p represents the main folder's name. I was thinking of somehow splitting path p into its subfolders and using parallel foreach loops to continue collecting file sizes; however, they have an unknown amount of subdirectories. This is where I am having problems with trying to get the folder size back. Thanks for the help in advance

Update

I call this function through this foreach loop below

           DirectoryInfo di = new DirectoryInfo    (Browse_Folders_Text_Box.Text);
            FileInfo[] parsedfilename = di.GetFiles("*.*", System.IO.SearchOption.TopDirectoryOnly);
            parsedfoldername = System.IO.Directory.GetDirectories(Browse_Folders_Text_Box.Text, "*.*", System.IO.SearchOption.TopDirectoryOnly);
            //parsedfilename = System.IO.Directory.GetDirectories(textBox1.Text, "*.*", System.IO.SearchOption.AllDirectories);





            // Process the list of folders found in the directory.

            type_label.Text = "Folder Names \n";


            List<string> NameList = new List<string>();
            foreach (string transfer2 in parsedfoldername)
            {

                this.Cursor = Cursors.WaitCursor;
                //Uses the path and takes the name from last folder used
                string dirName = new DirectoryInfo(@transfer2).Name;
                string dirDate = new DirectoryInfo(@transfer2).LastWriteTime.ToString();


                NameList.Add(dirName);
                //Form2 TextTable = new Form2(NameList.ToString());



                //Display_Rich_Text_Box.AppendText(dirName);
                //Display_Rich_Text_Box.AppendText("\n");
                Last_Date_Modified_Text_Box.AppendText(dirDate);
                Last_Date_Modified_Text_Box.AppendText("\n");


                try
                {
                    double b;

                    b = getDirectorySize(transfer2);
                    MetricByte(b);



                }
                catch (Exception)
                {
                    Size_Text_Box.AppendText("N/A \n");                      
                }

            }

            Display_Rich_Text_Box.Text = string.Join(Environment.NewLine, NameList);
            this.Cursor = Cursors.Default;

So what I was thinking when I thought of parallel foreach loops was to take the next instance names (subfolder name) that would be all on the same level and run them all at the same time with getDirectorySize() because I know there is at least 7 subfolders directly beneath the main folder name.

Answer 1

Parallel access to the same physical drive will not speed up the work.

Your main problem is the GetFiles method. It goes through all the subfolders collecting all file names. Then you pass in a loop on same files again.

Use the EnumerateFiles method instead.

Try this code. It will be much faster.

public long GetDirectorySize(string path)
{
    var dirInfo = new DirectoryInfo(path);
    long totalSize = 0;

    foreach (var fileInfo in dirInfo.EnumerateFiles("*.*", SearchOption.AllDirectories))
    {
        totalSize += fileInfo.Length;
    }
    return totalSize;
}

MSDN:

The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned; when you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient.