Run single test against multiple configurations in Visual Studio

Question

We have our integration tests set up using xUnit and Microsoft.AspNetCore.TestHost.TestServer to run tests against Web API running on ASP.NET Core 2.2.

Our Web API is a single code base that would be deployed separately multiple times based on some configuration or application setting differences like country, currency, etc.

Below diagram tries to explain our deployment set up:

We want to ensure that our integration tests run against all the deployments.

For both deployments, X and X` the API endpoint, request, and response are absolutely same. Hence, We would like to avoid repeating ourselves when it comes to integration tests for each deployment.

Here is the sample code explaining our current test set up:

TestStartup.cs

public class TestStartup : IStartup
{
    public IServiceProvider ConfigureServices(IServiceCollection services)
    {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", false)
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        // Code to add required services based on configuration


        return services.BuildServiceProvider();
    }

    public void Configure(IApplicationBuilder app)
    {
        app.UseMvc();

        // Code to configure test Startup
    }
}

TestServerFixture.cs

public class TestServerFixture
{

    public TestServerFixture()
    {
        var builder = new WebHostBuilder().ConfigureServices(services =>
        {
            services.AddSingleton<IStartup>(new TestStartup());
        });

        var server = new TestServer(builder);
        Client = server.CreateClient();
    }

    public HttpClient Client { get; private set; }
}

MyTest.cs

public class MyTest : IClassFixture<TestServerFixture>
{
    private readonly TestServerFixture _fixture;

    public MyTest(TestServerFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
    {
        //...
    }
}

Now, I'm looking to run ItShouldExecuteTwice_AgainstTwoSeparateConfigurations test in class MyTest more than once against two different configurations/ app settings or in other words against two different test deployments within Visual Studio.

• I know, I should be able to achieve this using a combination of build configurations (like DEBUG_SETTING1, DEBUG_SETTING2) and preprocessor directive (#if DEBUG_SETTING1).

• The other option could be to have a base test helper project with common methods and a separate integration project for each deployment.

Is there a better and more elegant way to achieve this?

Answer 1

Refactor the test startup to allow for it to be modified as needed for its test

For example

public class TestStartup : IStartup {
    private readonly string settings;

    public TestStartup(string settings) {
        this.settings = settings;
    }

    public void ConfigureServices(IServiceCollection services) {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile(settings, false) //<--just an example
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        //...Code to add required services based on configuration

    }

    public void Configure(IApplicationBuilder app) {
        app.UseMvc();

        //...Code to configure test Startup
    }
}

And have that pattern filter up through the fixture

public class TestServerFixture {
    static readonly Dictionary<string, TestServer> cache = 
        new Dictionary<string, TestServer>();

    public TestServerFixture() {
        //...
    }

    public HttpClient GetClient(string settings) {
        TestServer server = null;
        if(!cache.TryGetValue(settings, out server)) {
            var startup = new TestStartup(settings); //<---
            var builder = new WebHostBuilder()
                .ConfigureServices(services => {
                    services.AddSingleton<IStartup>(startup);
                });
            server = new TestServer(builder);
            cache.Add(settings, server);
        }
        return server.CreateClient();
    }
}

And eventually the test itself

public class MyTest : IClassFixture<TestServerFixture> {
    private readonly TestServerFixture fixture;

    public MyTest(TestServerFixture fixture) {
        this.fixture = fixture;
    }

    [Theory]
    [InlineData("settings1.json")]
    [InlineData("settings2.json")]
    public async Task Should_Execute_Using_Configurations(string settings) {
        var client = fixture.CreateClient(settings);

        //...use client

    }
}

Answer 2

@Nkosi's post fits very well with our scenario and my asked question. It's a simple, clean and easy to understand approach with maximum reusability. Full marks to the answer.

However, there were a few reasons why I could not go forward with the approach:

• In the suggested approach we couldn't run tests for only one particular setting. The reason it was important for us as in the future, there could two different teams maintaining their specific implementation and deployment. With Theory, it becomes slightly difficult to run only one setting for all the tests.

• There is a high probability that we may need two separate build and deployment pipelines for each setting/ deployment.

• While the API endpoints, Request, and Response are absolutely the same today, we do not know if it will continue to be the case as our development proceed.

Due to the above reasons we also considered the following two approaches:

Approach 1

Have a common class library which has common Fixture and Tests as abstract class

• Project Common.IntegrationTests

TestStartup.cs

public abstract class TestStartup : IStartup
{
    public abstract IServiceProvider ConfigureServices(IServiceCollection services);

    public void Configure(IApplicationBuilder app)
    {
        app.UseMvc();

        // Code to configure test Startup
    }
}

TestServerFixture.cs

public abstract class TestServerFixture
{

    protected TestServerFixture(IStartup startup)
    {
        var builder = new WebHostBuilder().ConfigureServices(services =>
        {
            services.AddSingleton<IStartup>(startup);
        });

        var server = new TestServer(builder);
        Client = server.CreateClient();
    }

    public HttpClient Client { get; private set; }
}

MyTest.cs

public abstract class MyTest
{
    private readonly TestServerFixture _fixture;

    protected MyTest(TestServerFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public void ItShouldExecuteTwice_AgainstTwoSeparateConfigurations()
    {
        //...
    }
}

• Project Setting1.IntegrationTests (References Common.IntegrationTests)

TestStartup.cs

public class TestStartup : Common.IntegrationTests.TestStartup
{
    public override IServiceProvider ConfigureServices(IServiceCollection services)
    {
       var configuration = new ConfigurationBuilder()
           .SetBasePath(Directory.GetCurrentDirectory())
           .AddJsonFile("appsettings.json", false) // appsettings for Setting1
           .AddEnvironmentVariables()
           .Build();

        services.AddMvc()
            .SetCompatibilityVersion(version: CompatibilityVersion.Version_2_2);

        // Code to add required services based on configuration


        return services.BuildServiceProvider();
    }
}

TestServerFixture.cs

public class TestServerFixture : Fixtures.TestServerFixture
{
    public TestServerFixture() : base(new TestStartup())
    {
    }
}

MyTests.cs

public class MyTests : Common.IntegrationTests.MyTests, IClassFixture<TestServerFixture>
{
    public MyTests(TestServerFixture fixture) : base(fixture)
    {
    }
}

• Project Setting2.IntegrationTests (References Common.IntegrationTests)

A similar structure as Setting1.IntegrationTests

This approach provided a good balance of reusability and flexibility to run/ modify the tests independently. However, I was still not 100% convinced with this approach as it meant for each common Test class we would need to have an implementation where we are not doing anything other than calling the base constructor.

Approach 2

In the second approach, we took the Approach 1 further and try to fix the issue we had with Approach 1 with Shared Project. From the documentation:

Shared Projects let you write common code that is referenced by a number of different application projects. The code is compiled as part of each referencing project and can include compiler directives to help incorporate platform-specific functionality into the shared code base.

Shared Project gave us the best of both worlds without the ugliness of link files and unnecessary class inheritance or abstraction. Our new set up is as follows:

Edit: I wrote a blog post on this where I have talked about our use-case and the solution in detail. Here is the link:

https://ankitvijay.net/2020/01/04/running-an-asp-net-core-application-against-multiple-db-providers-part-2/