xunit programmatically add new tests/"[Facts]"?

Question

We have a folder full of JSON text files that need to be set to a single URI. Currently it's all done with a single xUnit "[Fact]" as below

[Fact]
public void TestAllCases()
{
    PileOfTests pot = new PileOfTests();
    pot.RunAll();
}

pot.RunAll() then parses the folder, loads the JSON files (say 50 files). Each is then hammered against the URI to see is each returns HTTP 200 ("ok"). If any fail, we're currently printing it as a fail by using

System.Console.WriteLine("\n >> FAILED ! << " + testname + "\n");

This does ensure that failures catch our eye but xUnit thinks all tests failed (understandably). Most importantly, we can't specify to xunit "here, run only this specific test". It's all or nothing the way it's currently built.

How can I programmatically add test cases? I'd like to add them when I read the number and names of the *.json files.

Answer 1

The simple answer is: No, not directly. But there exists an, albeit a bit hacky, workaround, which is presented below.

Current situation (as of xUnit 1.9.1)

By specifiying the [RunWith(typeof(CustomRunner))] on a class, one can instruct xUnit to use the CustomRunner class - which must implement Xunit.Sdk.ITestClassCommand - to enumerate the tests available on the test class decorated with this attribute.

But unfortunately, while the invocation of test methods has been decoupled from System.Reflection + the actual methods, the way of passing the tests to run to the test runner haven't.

Somewhere down in the xUnit framework code for invoking a specific test method, there is a call to typeof(YourTestClass).GetMethod(testName).

This means that if the class implementing the test discovery returns a test name that doesn't refer to a real method on the test class, the test is shown in the xUnit GUI - but any attempts to run / invoke it end up with a TargetInvocationException.

Workaround

If one thinks about it, the workaround itself is relatively straightforward. A working implementation of it can be found here.

The presented solution first reads in the names of the files which should appear as different tests in the xUnit GUI. It then uses System.Reflection.Emit to dynamically generate an assembly with a test class containing a dedicated test method for each of the input files.

The only thing that each of the generated methods does is to invoke the RunTest(string fileName) method on the class that specified the [EnumerateFilesFixture(...)] attribute. See linked gist for further explanation.

Hope this helps; feel free to use the example implementation if you like.