Running a BackgroundWorker within another BackgroundWorker

Question

I'm attempting a very DB intensive task in a project. Here is a walk-through:

We need to search our DB of workers, we called Locums, and find one for a specific job. This procedure starts when we decide to process x number of jobs. So, on the click of a button, we process using the ProcessJobBatch() method. However, this method only process against a very limited number of Locums. So it takes less then 10 seconds to fill up a scheduler control. Now, once the limited number of Locums are served, we need to run a background task to check the rest of the Locums. There are around 1250 of them!

So, once ProcessJobBatch() finishes, a BackgroundWorker, BackgroundWorkerMoreLocums, goes off. Now, this worker basically does a simple loop: For each job, go through the whole 1250 employees. This takes way too long. I need to plan this out using an alternate strategy that I can't of ATM or I need to show a secondary progress bar for the inner for-each loop.

More Explanation: We import a batch of Jobs (10 to 70) numerous times on daily bases. Once a batch is imported, the application instructs the logged-in user to "Preference Find" those newly created jobs. The user already has a list of his favorite locums (1 to 20). He wants to distribute the jobs among his favorites first. That is done through ProcessJobBatch(). But, there are two scenarios that prevent the flow there and then:

• What if certain jobs didn't fall to any favorite locum?
• What if there is a locum in the whole DB who can do almost all the jobs but since he isn't favorite?

So, I end up with a scenario of matching a job with each Locum.

Question: Can second BackgroundWorker run within a BackgroundWorker's DoWork? Am I doing the second scan wrong?

Environment: Windows 7 Pro 64-bit, Visual Studio 2010, C#, .NET 4.0, and Windows Forms

    private void ButtonPreferenceFind_Click(object sender, EventArgs e) {
        if (LookUpBatches.EditValue != null) {
            JobBatch JobBatchSelected = DbContext.JobBatches.FirstOrDefault(job_batch=> job_batch.OID == LookUpBatches.EditValue.ToString());

            if (JobBatchSelected != null && JobBatchSelected.Jobs.Count(condition => condition.JobStatusID == 1) > 0) {
                if (XtraMessageBox.Show(String.Format("Are you sure to process {0} job(s)?", JobBatchSelected.Jobs.Count(condition => condition.JobStatusID == 1)), Text, MessageBoxButtons.YesNo, MessageBoxIcon.Question) == DialogResult.Yes) {
                    ProcessJobBatch(JobBatchSelected);

                    IEnumerable<Job> SpecificJobs = from req_jobs in JobBatchSelected.Jobs
                                                                                    where req_jobs.JobStatusID == 1
                                                                                    select req_jobs;

                    ProgressBarControlPreferenceFinder.EditValue = 0;
                    ProgressBarControlPreferenceFinder.Properties.Minimum = 0;
                    ProgressBarControlPreferenceFinder.Properties.Maximum = SpecificJobs.Count() - 1;

                    BackgroundWorkerMoreLocums.RunWorkerAsync(SpecificJobs);

                } else {
                    LookUpBatches.Focus();
                }

            } else {
                XtraMessageBox.Show("Unable to retrieve the selected batch or the batch has no processable jobs.", Text, MessageBoxButtons.OK, MessageBoxIcon.Exclamation);
                LookUpBatches.Focus();

            }

        } else {
            XtraMessageBox.Show("Select a batch first.", Text, MessageBoxButtons.OK, MessageBoxIcon.Exclamation);                   
            LookUpBatches.Focus();

        }
    }

    #region Background Searching
    private void BackgroundWorkerMoreLocums_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e) {
        try {
            e.Result = GetTableData(e.Argument);

        }
        catch (Exception ex) {
            XtraMessageBox.Show("Background Error: " + ex.Message, "Excite Engine 2", MessageBoxButtons.OK, MessageBoxIcon.Error);
            e.Result = ex;
        }
    }

    private void BackgroundWorkerMoreLocums_ProgressChanged(object sender, System.ComponentModel.ProgressChangedEventArgs e) {
        // only display progress, do not assign it to grid          
        ProgressBarControlPreferenceFinder.Increment(e.ProgressPercentage);
    }

    private void BackgroundWorkerMoreLocums_RunWorkerCompleted(object sender, System.ComponentModel.RunWorkerCompletedEventArgs e) {
        if (e.Result is DataTable) {
            //dataGridView1.DataSource = e.Result as DataTable;
        }
        else if (e.Result is Exception) {
        }
    }

    private DataTable GetTableData(Object JobList) {
        DataTable ResultDataTable = new DataTable();
        ResultDataTable.Columns.Add();


        IEnumerable<Job> JobBatchSelected = (IEnumerable<Job>)JobList;

        IEnumerable<Locum> LeftOverLocums = from lefties in DbContext.Locums
                                                                                //where SchedulerMatrixStorage.Resources.Items.Select(res => (long)res.Id).ToList().Contains(lefties.OID) == false
                                                                                select lefties;

        int NumOfJobsProcessed = 0;

        List<KeyValuePair<long, TemporaryPreferenceFindLocum>> AlreadyPrefferedLocums = new List<KeyValuePair<long, TemporaryPreferenceFindLocum>>();

        foreach (Job oneJob in JobBatchSelected) {

            foreach (Locum oneLocum in LeftOverLocums) {

                if (DbContext.Availabilities.Any(check => check.LocumID == oneLocum.OID && check.AvailableDate == oneJob.JobDate && check.AvailabilityStatusID == 1)) {
                    //This Locum can do this job

                    //Now check if he/she has been just alloted
                    if (AlreadyPrefferedLocums.Any(search => search.Key == oneLocum.OID && search.Value.JobDate == oneJob.JobDate) == false) {
                        //No? Cool!                     
                        //Add to the list to prevent double allocation
                        AlreadyPrefferedLocums.Add(new KeyValuePair<long, TemporaryPreferenceFindLocum>(oneJob.OID, new TemporaryPreferenceFindLocum(oneJob.JobDate, oneJob.OID, oneLocum.OID, oneLocum.FirstName + " " + oneLocum.LastName)));

                    }
                    else {
                        continue;

                    }

                }
                else {
                    //Not marked as Avaliable on the required job date...
                    continue;

                }

            }

            NumOfJobsProcessed++;
            BackgroundWorkerMoreLocums.ReportProgress((int)(NumOfJobsProcessed * 100F / (JobBatchSelected.Count() - 1)));
        }

        return ResultDataTable;
    }
    #endregion

Answer 1

A BackgroundWorker can be started from within the DoWork handler of another BackgroundWorker, but you need to be aware of the consequences of using such a scheme. When you start a background worker from your main UI thread the DoWork handler is executed on a thread pool thread while the ProgressChanged and RunWorkerCompleted are executed back on the main UI thread making it safe for you to interact with windows forms controls.

This scenario is guaranteed when you start the worker from the main UI thread because it picks up the SynchronizationContext available on that thread and which is initialized by the windows forms infra-structure.

However, when you start a background worker from the DoWork handler of another worker, you'll be starting it from a thread pool thread that lacks the synchronization context causing the ProgressChanged and RunWorkerCompleted handlers to also be executed on thread pool threads and not in your main UI thread making it unsafe for you to interact with windows forms controls from within those handlers.

Answer 2

It is quite common to have one background thread spawn new background threads. I don't think it is a problem if you scan the list on a background thread and process each list item on another thread.

In such cases there is no background worker within another. There is just a background worker starting other threads.

Things you should consider -

1. Be aware of what you do in the completed event handlers in case you handle that event.

2. Consider the performance implications of running so many threads for small tasks. You should consider using PLINQ or parallel tasks so that .Net can handle the partitioning of input and merging of results giving you optimum performance.