How to make xUnit to run a Theory parallel?

Question

I've got a test (Theory) which is slow and a bunch of test cases for it. So I want them to run simultaneously.

I've created a simple example:

[Theory]
[MyTestData]
public void MyTheory(int num, int sleep)
{
    Console.WriteLine("{0:HH:mm:ss.ffff} - Starting {1} - Sleeping {2}", DateTime.Now, num, sleep);
    Thread.Sleep(sleep);
    Console.WriteLine("{0:HH:mm:ss.ffff} - Finished {1} - Sleeping {2}", DateTime.Now, num, sleep);
}

[AttributeUsage(AttributeTargets.Method)]
public class MyTestDataAttribute : DataAttribute
{
    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
        yield return new object[2] { 1, 5000 };
        yield return new object[2] { 2, 2000 };
        yield return new object[2] { 3, 4000 };
    }
}

The command line to run the test:

"\packages\xunit.runner.console.2.0.0\tools\xunit.console" "\Projects\xUnitTest\xUnitTest\bin\Debug\xUnitTest.dll" -parallel all > xUnitResult.txt

And the result:

xUnit.net console test runner(64-bit.NET 4.0.30319.42000)
Copyright(C) 2015 Outercurve Foundation.

Discovering: xUnitTest
Discovered:  xUnitTest
Starting:    xUnitTest
21:55:39.0449 - Starting 2 - Sleeping 2000
21:55:41.0627 - Finished 2 - Sleeping 2000
21:55:41.0783 - Starting 1 - Sleeping 5000
21:55:46.0892 - Finished 1 - Sleeping 5000
21:55:46.0892 - Starting 3 - Sleeping 4000
21:55:50.0989 - Finished 3 - Sleeping 4000
Finished:    xUnitTest

=== TEST EXECUTION SUMMARY ===
   xUnitTest Total: 3, Errors: 0, Failed: 0, Skipped: 0, Time: 11,137s

Which is pretty serial. I'm sure it is possible to make it parallel.

Answer 1

This is not currently possible as of xUnit 2.1. According to the parallelization docs,

By default, each test class is a unique test collection. Tests within the same test class will not run in parallel against each other.

What the docs don't make explicitly clear is:

• the smallest "parallelizable" unit in xUnit is a collection
• there's no way for tests in the same class to be in different collections

By extension, it's impossible to have parallelized theories because theories can't be split across multiple classes.

---

In your case, this means you have at least two options:

• Refactor your code under test so that whatever is taking so much time is abstracted away. For example, let's say you have to run some business logic and then make a database call. If you can separate the testable business logic into another class, you can run your theory against that (not in parallel, but in <1 ms), and separately test the slow data access code.

• If you have sufficiently few test cases, just make a new class per test case and use Fact instead of Theory. You can even put them all in a single file. It's more verbose, and you lose the "cool" factor of using theories, but you'll get parallel execution.

Answer 2

Although not directly possible with xUnit, you can work around that if you need to. With drawbacks, like you have to manually define the number of parallel executions via classes, so if you want to parallelize it over two threads, you need to create two classes.

public abstract class ParellelTheoryBase
{
    public static List<int> testValues = new List<int> {1, 2, 3, 4, 5, 6};
}

public class ParallelTheory_1of2 : ParellelTheoryBase
{
    public static List<object[]> theoryData = testValues.Where((data, index) => index % 2 == 0).Select(data => new object[] { data }).ToList();

    [Theory]
    [MemberData(nameof(theoryData))]
    public void DoSomeLongRunningAddition(int data)
    {
        Assert.True(data < 7);
        Thread.Sleep(5000);
    }
}

public class ParallelTheory_2of2 : ParellelTheoryBase
{
    public static List<object[]> theoryData = testValues.Where((data, index) => index % 2 == 1).Select(data => new object[] { data }).ToList();

    [Theory]
    [MemberData(nameof(theoryData))]
    public void DoSomeLongRunningAddition(int data)
    {
        Assert.True(data < 7);
        Thread.Sleep(5000);
    }
}

In this example, I've defined the property in ParellelTheoryBase, which is a base class for the actual test classes. Then, ParallelTheory_1of2 and ParallelTheory_2of2 inherit from that class to have access to the testValues. Now, the Theory uses this theoryData for test execution, and it selects only odd or even (index % 2 == 1 or index % 2 == 0) data from the list.

It gets picked up in the Visual Studio test explorer:

And runs in parallel:

[xUnit.net 00:00:00.3397963]   Starting
[xUnit.net 00:00:15.5696617]   Finished

This might be a solution where it's not easy to know in beforehand how much test data you have. I use that, for example, in integration testing a file parser where the input is any file within a directory and new added files will automatically be picked up for the tests.

In your example, I think you could easily modify the MyTestData attribute to accept parameters for totalCountParallels and currentIndex.