How exactly do MbUnit's [Parallelizable] and DegreeOfParallelism work?

Question

I thought I understood how MbUnit's parallel test execution worked, but the behaviour I'm seeing differs sufficiently much from my expectation that I suspect I'm missing something!

I have a set of UI tests that I wish to run concurrently. All of the tests are in the same assembly, split across three different namespaces. All of the tests are completely independent of one another, so I'd like all of them to be eligible for parallel execution.

To that end, I put the following in the AssemblyInfo.cs:

[assembly: DegreeOfParallelism(8)]

[assembly: Parallelizable(TestScope.All)]

My understanding was that this combination of assembly attributes should cause all of the tests to be considered [Parallelizable], and that the test runner should use 8 threads during execution. My individual tests are marked with the [Test] attribute, and nothing else. None of them are data-driven.

However, what I actually see is at most 5-6 threads being used, meaning that my test runs are taking longer than they should be.

Am I missing something? Do I need to do anything else to ensure that all of my 8 threads are being used by the runner?

N.B. The behaviour is the same irrespective of which runner I use. The GUI, command line and TD.Net runners all behave the same as described above, again leading me to think I've missed something.

EDIT: As pointed out in the comments, I'm running v3.1 of MbUnit (update 2 build 397). The documentation suggests that the assembly level [parallelizable] attribute is available, but it does also seem to reference v3.2 of the framework despite that not yet being available.

EDIT 2: To further clarify, the structure of my assembly is as follows:

assembly
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)
- namespace
   - fixture
      - tests (each carrying only the [Test] attribute)
   - fixture
      - tests (each carrying only the [Test] attribute)

EDIT 3: OK, I've now noticed that if I only ever run one fixture at a time, the maximum number of tests running concurrently is always 8. As soon as I select multiple fixtures, it drops to either 5 or 6. If I take the contents of two fixtures (currently they contain 12 tests each) and drop them into the same fixture (for a total of 24 tests in that one fixture) that fixture will also always run 8 tests concurrently.

This seems to show that it isn't an issue in the individual tests, but rather in how the assembly level attributes percolate down to the fixture, or how the test runner consumes those attributes.

Additionally, I also observed (when running two fixtures) that once one of the two fixtures had been executed in its entirety, the runner starts to execute more tests concurrently when its back down to running only one fixture. For me right now, the first fixture gets done executing when there are 7 tests left to run in the second fixture. As soon as that happens, the number of tests running concurrently jumps up from the previous 5 or 6 to the maximum available of 7.

Answer 1

According to the release note of Gallio v3.0.6:

MbUnit helps you get the most out of your multi-core CPU. Mark any test [Parallelizable] and it will be permitted to run in parallel with other parallelizable tests in the same fixture.

Fixtures can also be marked parallelizable to enable them to be run in parallel with other parallelizable fixtures.

Please note that if you want all tests within a fixture to be considered parallelizable then you still need to add [Parallelizable] to each of them. (We might add a feature to set this at the fixture or assembly level later based on user feedback.)

Also note that just because a test or fixture is marked parallelizable does not mean it will run in parallel with other tests in particular. For the sake of efficiency, we limit the number of active tests threads based on the configured degree of parallelism. If you want a specific number of instances of a test to run in parallel with each other, consider using [ThreadedRepeat].

The degree of parallelism setting controls the maximum number of tests that MbUnit will attempt to run in parallel with one another. By default, the degree of parallelism equals the number of CPUs you have, or 2 at a minimum.

If you don't like the default then you can override the degree of parallelism at the assembly-level like this:

I don't know if it helps. Maybe Jeff could give more details as he had implemented that feature.