Why is a .NET unit test covering a Parallel.foreach loop hardware-dependent?

Question

I'm using Moq to unit test some code that includes a Parallel.foreach loop.

The Arrange phase sets up 4 exceptions to be thrown within the loop and then wrapped in an AggregateException.

This passed on my i7 processor and I checked in the code.

Afterwards, a colleague complained it wasn't passing for him. It turned out Parallel.foreach was only spawning 2 threads on his Core2duo before bombing out and hence only 2 exceptions were wrapped in the AggregateException.

The question is what to do about this so the unit test doesn't depend on processor architecture? A couple of thoughts:-

1. There is a Microsoft article on manually adding exceptions to the AggregateException but we're not keen on doing this as the loop should exit at the earliest opportunity if there is a problem.
2. ParallelOptions.MaxDegreeOfParallelism can put an upper limit on the number of threads used. But unless this is turned down to 1 (which seems more like cheating than proper unit testing) how can the unit test know how many threads will actually be used and hence set up the Arrange and Assert phases propertly?

Answer 1

You should not test something like that - it's implementation detail.

Thing is - Parallel.ForEach will process elements until it gets exception. When exception occurs it will stop processnig any new elements (but will finish processing of the ones currently processed) and then throw AgregateException.

Now - your i7 CPU have 4 cores + Hyper Threading, which results in more threads spawned for processing, thus you can get more exceptions (because for example 4 things can be processed at the same time when exception occurs). But on Core2Duo with just 2 cores, only 2 items will be processed at the same time (it's because TPL is smart enough to create only enough threads for processing, not more than available cores).

Testing that actually 4 exception occured gives you no knowledge. It's machine dependent. You should instead test if at least one exception occured - since that's what you are expecting. If future user will run your code on old single core machine he will recieve just this one exception in AggregateException.

Number of thrown exception is machine specific, like for example calculation time - you would not assert on how long something was calculated, so you should not assert on number of exception in this case.

Answer 2

Unit test verifies that what you're expecting to happen and what actually happens are the same thing. That means you need to ask yourself what are you expecting to happen.

If you expect that all potential errors in processing will be reported, then your colleague's run actually found a bug in your code. This mean that plain Parallel.ForEach() won't work for you, but something like Parallel.ForEach() with a try-catch and custom exception processing would.

If what you expect is that each thrown exception will be caught, then that's what you need to test. This may complicate your test or it may not be possible to test, depending on how are you throwing the exceptions. A simple test for this would look like this:

[Test]
public void ParallelForEachExceptionsTest()
{
    var exceptions = new ConcurrentQueue<Exception>();
    var thrown = Assert.Throws<AggregateException>(
        () => Parallel.ForEach(
            Enumerable.Range(1, 4), i =>
            {
                var e = new Exception(i.ToString());
                exceptions.Enqueue(e);
                throw e;
            }));

    CollectionAssert.AreEquivalent(exceptions, thrown.InnerExceptions);
}

If what you expect is that when one or more exceptions are thrown, then at least some of them will be caught, then that's what you should test, and you shouldn't worry if all the exceptions were caught correctly.