C# Multiple BackgroundWorkers

Question

I am trying to setup multiple BackgroundWorkers to do work and when not busy start doing the next bit of work. I can't seem to get them working properly. I have the below code.

When I set FilesToProcess equal or less than MaxThreads it works perfectly although if I make it higher, the app freezes.

I am sure it is something simple, but I just can't see it.

using System;
using System.ComponentModel;
using System.Threading;
using System.Windows.Forms;

namespace bgwtest
{
    public partial class Form1 : Form
    {
        private const int MaxThreads = 20;
        private const int FilesToProcess = 21;
        private BackgroundWorker[] threadArray = new BackgroundWorker[MaxThreads];

        public Form1()
        {
            InitializeComponent();
        }

        private void Form1Load(object sender, EventArgs e)
        {
            InitializeBackgoundWorkers();
        }

        private void InitializeBackgoundWorkers()
        {
            for (var f = 0; f < MaxThreads; f++)
            {
                threadArray[f] = new BackgroundWorker();
                threadArray[f].DoWork += new DoWorkEventHandler(BackgroundWorkerFilesDoWork);
                threadArray[f].RunWorkerCompleted += new RunWorkerCompletedEventHandler(BackgroundWorkerFilesRunWorkerCompleted);
                threadArray[f].WorkerReportsProgress = true;
                threadArray[f].WorkerSupportsCancellation = true;
            }
        }

        private void button1_Click(object sender, EventArgs e)
        {
            for (var f = 0; f < FilesToProcess; f++)
            {
                var fileProcessed = false;
                while (!fileProcessed)
                {
                    for (var threadNum = 0; threadNum < MaxThreads; threadNum++)
                    {
                        if (!threadArray[threadNum].IsBusy)
                        {
                            Console.WriteLine("Starting Thread: {0}", threadNum);

                            threadArray[threadNum].RunWorkerAsync(f);
                            fileProcessed = true;
                            break;
                        }
                    }
                    if (!fileProcessed)
                    {
                        Thread.Sleep(50);
                    }
                }
            }
        }

        private void BackgroundWorkerFilesDoWork(object sender, DoWorkEventArgs e)
        {
            ProcessFile((int)e.Argument);

            e.Result = (int)e.Argument;
        }

        private static void ProcessFile(int file)
        {
            Console.WriteLine("Processing File: {0}", file);
        }

        private void BackgroundWorkerFilesRunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            if (e.Error != null)
            {
                MessageBox.Show(e.Error.Message);
            }

            Console.WriteLine("Processed File: {0}", (int)e.Result);
        }
    }
}

Answer 1

The problem appears to be that your workers are never completing. Why this is, I'm not sure; it has something to do with the fact that the method (and thread) you are running them from is not itself completing. I was able to solve the problem by creating another worker to assign files to the worker array:

    private BackgroundWorker assignmentWorker;

    private void InitializeBackgoundWorkers() {
        assignmentWorker = new BackgroundWorker();
        assignmentWorker.DoWork += AssignmentWorkerOnDoWork;
        // ...
    }

    private void AssignmentWorkerOnDoWork( object sender, DoWorkEventArgs doWorkEventArgs ) {
        for( var f = 0; f < FilesToProcess; f++ ) {
            var fileProcessed = false;
            while( !fileProcessed ) {
                for( var threadNum = 0; threadNum < MaxThreads; threadNum++ ) {
                    if( !threadArray[threadNum].IsBusy ) {
                        Console.WriteLine( "Starting Thread: {0}", threadNum );

                        threadArray[threadNum].RunWorkerAsync( f );
                        fileProcessed = true;
                        break;
                    }
                }
                if( !fileProcessed ) {
                    Thread.Sleep( 50 );
                    break;
                }
            }
        }
    }

    private void button1_Click( object sender, EventArgs e ) {
        assignmentWorker.RunWorkerAsync();
    }

I'm not happy with this answer because I don't know why, exactly, it didn't work as you originally designed it. Perhaps someone else can answer that...? At least this will get you a working version.

EDIT: Your original version didn't work because the BackgroundWorkerFilesRunWorkerCompleted runs on the same thread as button1_Click (the UI thread). Since you are not freeing up the UI thread, the thread is never getting marked as complete.

Answer 2

In completion to Ethan Brown's answer, the problem effectively comes from the fact that you are running a forever busy loop inside the UI thread.

If filesToProcess is lower than the max number of threads, the loop will assign each file to one thread and exit. No problems there.

But if you have more files than threads, then look at what happens when you try process the file while all threads are already running :

• you check if all threads are busy, yes they are.
• You wait by doing a thread sleep.

The problem is that when a background worker completes, it will post a message into the UI Thread message queue. And your UI thread is actually busy checking your threads and waiting.

To solve this :

• Either you put the loop in another thread (as Ethan Brown's solution)
• Eihter you add Application.DoEvents() before or after the Thread.Sleep() in order to force your UI thread to process the messages. It is not really good practice, but illustrates very well the problem here.