xUnit Non-Static MemberData

Question

I have the following DatabaseFixture which has worked well for all tests I have created up to this point. I use this fixture for integration tests so I can make real assertions on database schema structures.

public class DatabaseFixture : IDisposable
{
    public IDbConnection Connection => _connection.Value;
    private readonly Lazy<IDbConnection> _connection;

    public DatabaseFixture()
    {
        var environment = Environment.GetEnvironmentVariable("ASPNET_ENVIRONMENT") ?? "Development";
        var configuration = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("AppSettings.json", optional: false, reloadOnChange: true)
            .AddJsonFile($"AppSettings.{environment}.json", optional: true, reloadOnChange: true)
            .Build();

        _connection = new Lazy<IDbConnection>(() =>
        {
            var connection = new MySqlConnection(configuration["ConnectionStrings:MyDatabase"]);
            connection.Open();
            return connection;
        });
    }

    public void Dispose()
    {
        Connection?.Dispose();
    }
}

[CollectionDefinition("Database Connection Required")]
public class DatabaseConnectionFixtureCollection : ICollectionFixture<DatabaseFixture>
{
}

The problem I am facing is I now need to invoke a test method like MyDataIsAccurate(...) with each record from a table in the database. xUnit offers the [MemberData] attribute which is exactly what I need but it requires a static enumerable set of data. Does xUnit offer a clean way of sharing my DatabaseFixture connection instance statically or do I just need to suck it up and expose a static variable of the same connection instance?

[Collection("Database Connection Required")]
public class MyTests
{
    protected DatabaseFixture Database { get; }

    // ERROR: Can't access instance of DatabaseFixture from static context...
    public static IEnumerable<object[]> MyData => Database.Connection.Query("SELECT * FROM table).ToList();

    public MyTests(DatabaseFixture databaseFixture)
    {
        Database = databaseFixture;
    }

    [Theory]
    [IntegrationTest]
    [MemberData(nameof(MyData))]
    public void MyDataIsAccurate(int value1, string value2, string value3)
    {
        // Assert some stuff about the data...
    }
}

Answer 1

You cannot access the fixture from the code that provides the test cases (whether that is a MemberData property or a ClassData implementation or a custom DataAttribute subclass.

Reason

Xunit creates an AppDomain containing all the data for the test cases. It builds up this AppDomain with all of those data at the time of test discovery. That is, the IEnumerable<object[]>s are sitting in memory in the Xunit process after the test assembly is built, and they are sitting there just waiting for the tests to be run. This is what enables different test cases to show up as different tests in test explorer in visual studio. Even if it's a MemberData-based Theory, those separate test cases show up as separate tests, because it's already run that code, and the AppDomain is standing by waiting for the tests to be run. On the other hand, fixtures (whether class fixtures or collection fixtures) are not created until the test RUN has started (you can verify this by setting a breakpoint in the constructor of your fixture and seeing when it is hit). This is because they are meant to hold things like database connections that shouldn't be left alive in memory for long periods of time when they don't need to be. Therefore, you cannot access the fixture at the time the test case data is created, because the fixture has not been created.

If I were to speculate, I would guess that the designers of Xunit did this intentionally and would have made it this way even if the test-discovery-loads-the-test-cases-and-therefore-must-come-first thing was not an issue. The goal of Xunit is not to be a convenient testing tool. It is to promote TDD, and a TDD-based approach would allow anyone to pick up the solution with only their local dev tools and run and pass the same set of tests that everyone else is running, without needing certain records containing test case data to be pre-loaded in a local database.

Note that I'm not trying to say that you shouldn't do what you're trying, only that I think the designers of Xunit would tell you that your test cases and fixtures should populate the database, not the other way around. I think it's at least worth considering whether that approach would work for you.

Workaround #1

Your static database connection may work, but it may have unintended consequences. That is, if the data in your database changes after the test discovery is done (read: after Xunit has built up the test cases) but before the test itself is run, your tests will still be run with the old data. In some cases, even building the project again is not enough--it must be cleaned or rebuilt in order for test discovery to be run again and the test cases be updated.

Furthermore, this would kind of defeat the point of using an Xunit fixture in the first place. When Xunit disposes the fixture, you are left with the choice to either: dispose the static database connection (but then it will be gone when you run the tests again, because Xunit won't necessarily build up a new AppDomain for the next run), or do nothing, in which case it might as well be a static singleton on some service locator class in your test assembly.

Workaround #2

You could parameterize the test with data that allows it to go to the fixture and retrieve the test data. This has the disadvantage that you don't get the separate test cases listed as separate tests in either test explorer or your output as you would hope for with a Theory, but it does load the data at the time of the tests instead of at setup and therefore defeats the "old data" problem as well as the connection lifetime problem.

Summary

I don't think such a thing exists in Xunit. As far as I know, your options are: have the test data populate the database instead of the other way around, or use a never-disposed static singleton database connection, or pull the data in your test itself. None of these are the "clean" solution you were hoping for, but I doubt you'll be able to get much better than one of these.

Answer 2

There is a way of achieving what you want, using delegates. This extremely simple example explains it quite well:

using System;
using System.Collections.Generic;

using Xunit;

namespace YourNamespace
{
    public class XUnitDeferredMemberDataFixture
    {
        private static string testCase1;
        private static string testCase2;

        public XUnitDeferredMemberDataFixture()
        {
            // You would populate these from somewhere that's possible only at test-run time, such as a db
            testCase1 = "Test case 1";
            testCase2 = "Test case 2";
        }

        public static IEnumerable<object[]> TestCases
        {
            get
            {
                // For each test case, return a human-readable string, which is immediately available
                // and a delegate that will return the value when the test case is run.
                yield return new object[] { "Test case 1", new Func<string>(() => testCase1) };
                yield return new object[] { "Test case 2", new Func<string>(() => testCase2) };
            }
        }

        [Theory]
        [MemberData(nameof(TestCases))]
        public void Can_do_the_expected_thing(
            string ignoredTestCaseName, // Not used; useful as this shows up in your test runner as human-readable text
            Func<string> testCase) // Your test runner will show this as "Func`1 { Method = System.String.... }"
        {
            Assert.NotNull(testCase);

            // Do the rest of your test with "testCase" string.
        }
    }
}

In the OP's case, you could access the database in the XUnitDeferredMemberDataFixture constructor.