Multiple Parallel.ForEach calls, MemoryBarrier?

Question

I have a bunch of data rows, and I want to use Parallel.ForEach to compute some value on each row like this...

class DataRow
{
    public double A { get; internal set; }
    public double B { get; internal set; }
    public double C { get; internal set; }

    public DataRow()
    {
        A = double.NaN;
        B = double.NaN;
        C = double.NaN;
    }
}

class Program
{
    static void ParallelForEachToyExample()
    {
        var rnd = new Random();
        var df = new List<DataRow>();

        for (int i = 0; i < 10000000; i++)
        {
            var dr = new DataRow {A = rnd.NextDouble()};
            df.Add(dr);
        }

        // Ever Needed? (I)
        //Thread.MemoryBarrier();

        // Parallel For Each (II)
        Parallel.ForEach(df, dr =>
        {
            dr.B = 2.0*dr.A;
        });

        // Ever Needed? (III)
        //Thread.MemoryBarrier();

        // Parallel For Each 2 (IV)
        Parallel.ForEach(df, dr =>
        {
            dr.C = 2.0 * dr.B;
        });
    }
}

(In this example, there's no need to parallelize and if there was, it could all go inside one Parallel.ForEach. But this is meant to be a simplified version of some code where it makes sense to set it up like this).

Is it possible for the reads to be re-ordered here so that I end up with a data row where B != 2A or C != 2B?

Say the first Parallel.ForEach (II) assigns worker thread 42 to work on data row 0. And the second Parallel.ForEach (IV) assigns worker thread 43 to work on data row 0 (as soon as the first Parallel.ForEach finishes). Is there a chance that the read of dr.B for row 0 on thread 43 returns double.NaN since it hasn't seen the write from thread 42 yet?

And if so, does inserting a memory barrier at III help at all? Would this force the updates from the first Parallel.ForEach to be visible to all threads before the second Parallel.ForEach starts?

Answer 1

The work started by a Parallel.ForEach() will be done before it returns. Internally, ForEach() spawns a Task for each iteration, and calls Wait() on each one. As a result, you do not need to synchronize access between ForEach() calls.

You do need to keep that in mind for individual tasks with ForEach() overloads that allow you access to loop state, aggregating results from tasks, etc. For example in this trivial example which sums up 1 ≤ x ≤ 100, the Action passed to localFinally of Parallel.For() has to be concerned about synchronization issues,

var total = 0;

Parallel.For(0, 101, () => 0,  // <-- localInit
(i, state, localTotal) => { // <-- body
  localTotal += i;
  return localTotal;
}, localTotal => { <-- localFinally
  Interlocked.Add(ref total, localTotal); // Note the use of an `Interlocked` static method
});

// Work of previous `For()` call is guaranteed to be done here

Console.WriteLine(total);

In your example, it is not necessary to insert a memory barrier between the ForEach() calls. Specifically, loop IV can depend on the results of II being completed, and Parallel.ForEach() already inserted III for you.

Snippet sourced from: Parallel Framework and avoiding false sharing