XUnit.net capture the result of each test right after it runs

Question

I am attempting to use XUnit.net as a replacement for a custom home-grown test scheduler. One of the features of the home grown scheduler is that for tests which are long-running, it outputs the result of the test (pass/fail, as well as the exception which caused the failure) to a database right after the test passes/fails.

Because there are a possible large number of long running tests this is useful in order to view test progress over the course of a run (don't have to wait for the full test pass until you see all the results, as the full test pass may take time).

The XUnit.net source is here: https://github.com/xunit/xunit

I've taken a look and see BeforeAfterTestAttribute, but the "After" method doesn't give access to the test results, just to the test method. I want something similar but which also gives access to the test result, so that I can report to a database the result immediately (instead of having to wait for the full test suite to finish).

It seems (from source) that the only thing that has access to the actual test result is the TestRunner, but as far as I can tell there is no extensibility model for the test runner.

One possible workaround I've come up with is the following:

[Fact]
TestMethod()
{
    //This method takes a lambda, handles exceptions, uploads to my
    //database, and then rethrows.
    RunTestWithExtraLogging(() => 
    {
        //Actual test goes here
    }
}

The above solution isn't ideal though as it requires the author of each test to call the "RunTestWithExtraLogging" method.

PS: I'm open to considering a different test framework (other than xUnit.net) if it supports this...

Answer 1

Yes, I believe you need to create you own runner. A simple example on how this can be done is provided below (you will need to add the logic you require in the TestMethodRunnerCallback method:

namespace MyTestRunner
{
    using System;
    using System.Linq;
    using Xunit;        

    public class Program
    {
        public static int Main(string[] args)
        {
            var testAssembly = TestAssemblyBuilder.Build(
                new ExecutorWrapper(args[0], null, false));

            var tests = testAssembly.EnumerateTestMethods(x =>  x
                .DisplayName
                .ToLowerInvariant();

            var testMethods = (args.Length > 1 && !string.IsNullOrEmpty(args[1])
                ? tests.Contains(args[1].ToLowerInvariant())).ToList()
                : tests.ToList();

            if (testMethods.Count == 0)
                return 0;

            var runnerCallback = new TestMethodRunnerCallback();
            testAssembly.Run(testMethods, runnerCallback);

            return runnerCallback.FailedCount;
        }

        public class TestMethodRunnerCallback : ITestMethodRunnerCallback
        {
            public int FailedCount { get; private set; }

            public void AssemblyFinished(TestAssembly testAssembly, int total, int failed, int skipped, double time)
            {
                FailedCount = failed;
            }

            public void AssemblyStart(TestAssembly testAssembly)
            {
            }

            public bool ClassFailed(TestClass testClass, string exceptionType, string message, string stackTrace)
            {
                return true;
            }

            public void ExceptionThrown(TestAssembly testAssembly, Exception exception)
            {
            }

            public bool TestFinished(TestMethod testMethod)
            {
                return true;
            }

            public bool TestStart(TestMethod testMethod)
            {
                return true;
            }
        }
    }
}

edit: It appears as though in xUnit.net 2.0 TestAssemblyBuilder has been replaced with XunitFrontController. The code below shows a snippet of how to capture the result of a test pass as the tests are running:

public class Program
{
    public static void Main(string[] args)
    {
        string assemblyPath = @"path";
        XunitFrontController controller = new XunitFrontController(
            assemblyPath);

        TestAssemblyConfiguration assemblyConfiguration = new TestAssemblyConfiguration();

        ITestFrameworkDiscoveryOptions discoveryOptions = TestFrameworkOptions.ForDiscovery(assemblyConfiguration);
        ITestFrameworkExecutionOptions executionOptions = TestFrameworkOptions.ForExecution(assemblyConfiguration);

        IMessageSink messageSink = new CustomTestMessageVisitor<ITestMessage>();

        Console.WriteLine("Running tests");

        controller.RunAll(
            messageSink: messageSink,
            discoveryOptions: discoveryOptions,
            executionOptions: executionOptions);
    }
}

public class CustomTestMessageVisitor<T> : TestMessageVisitor<T> where T : ITestMessage
{
    protected override bool Visit(ITestFinished message)
    {
        Console.WriteLine("Test {0} finished.",
            message.Test.DisplayName);
        return true;
    }

    protected override bool Visit(ITestPassed message)
    {
        Console.WriteLine("Test {0} passed", message.Test.DisplayName);
        return true;
    }

    protected override bool Visit(ITestFailed message)
    {
        StringBuilder stringBuilder = new StringBuilder();
        foreach (string exceptionType in message.ExceptionTypes)
        {
            stringBuilder.AppendFormat("{0}, ", exceptionType);
        }

        Console.WriteLine("Test {0} failed with {1}", 
            message.Test.DisplayName, 
            stringBuilder.ToString());
        return true;
    }
}