ASP.NET Core Integration Test works locally but throws null reference exception when running in production environment

Question

I have an ASP.NET Core 2.2 Razor Pages Web App that I have written some integration tests for following the official guide.

I can get the tests to run locally using dotnet test or the test runners built into Visual Studio. However, on the build server (Azure DevOps Hosted 2017 agent) the tests will return a 500 error. I thought it might be related to user secrets as stated on Scott Hanselman's guide but I am still getting the same error, even after implementing some of his suggested fixes (I don't believe I need all of them) :

• Added builder.AddUserSecrets<Startup>(); to Startup.
• Implemented a CustomWebApplicationFactory to set the environment to "Development" - code below has this as "Production" to reproduce the failure.

I also sanity checked against this guide which is more controller focused but since I only care about the response codes at this stage it serves my purpose. I have downloaded the verbose logs and they don't shed any light on the issue.

My code is below:

CustomWebApplicationFactory:

using Microsoft.AspNetCore.Authentication.AzureAD.UI;
using Microsoft.AspNetCore.Authentication.OpenIdConnect;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;

namespace WebPortal.Int.Tests
{
    /// <summary>
    /// Based on https://fullstackmark.com/post/20/painless-integration-testing-with-aspnet-core-web-api
    /// </summary>
    public class CustomWebApplicationFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class
    {
        public CustomWebApplicationFactory() { }

        protected override void ConfigureWebHost(IWebHostBuilder builder)
        {
            builder
                .ConfigureTestServices(
                    services =>
                    {
                        services.Configure(AzureADDefaults.OpenIdScheme, (System.Action<OpenIdConnectOptions>)(o =>
                        {
                            // CookieContainer doesn't allow cookies from other paths
                            o.CorrelationCookie.Path = "/";
                            o.NonceCookie.Path = "/";
                        }));
                    }
                )
                .UseEnvironment("Production")
                .UseStartup<Startup>();

        }
    }
}

AuthenticationTests:

using Microsoft.AspNetCore.Mvc.Testing;
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Xunit;

namespace WebPortal.Int.Tests
{
    public class AuthenticationTests : IClassFixture<CustomWebApplicationFactory<Startup>>
    {
        private HttpClient _httpClient { get; }

        public AuthenticationTests(CustomWebApplicationFactory<Startup> fixture)
        {
            WebApplicationFactoryClientOptions webAppFactoryClientOptions = new WebApplicationFactoryClientOptions
            {
                // Disallow redirect so that we can check the following: Status code is redirect and redirect url is login url
                // As per https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-2.2#test-a-secure-endpoint
                AllowAutoRedirect = false
            };

            _httpClient = fixture.CreateClient(webAppFactoryClientOptions);
        }

        [Theory]
        [InlineData("/")]
        [InlineData("/Index")]
        [InlineData("/Error")]
        public async Task Get_PagesNotRequiringAuthenticationWithoutAuthentication_ReturnsSuccessCode(string url)
        {
            // Act
            HttpResponseMessage response = await _httpClient.GetAsync(url);

            // Assert
            try
            {
                response.EnsureSuccessStatusCode();
            }
            catch (HttpRequestException ex)
            {
                Console.WriteLine(ex.Message, ex.InnerException.Message);
            }
        }
    }
}

Startup:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using WebPortal.Authentication;
using WebPortal.Common.ConfigurationOptions;
using WebPortal.DataAccess;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Authentication.AzureAD.UI;
using Microsoft.AspNetCore.Authentication.OpenIdConnect;
using Microsoft.AspNetCore.Http;

namespace WebPortal
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.Configure<CookiePolicyOptions>(options =>
            {
                // This lambda determines whether user consent for non-essential cookies is needed for a given request.
                options.CheckConsentNeeded = context => false;
                options.MinimumSameSitePolicy = SameSiteMode.None;
            });

            services.AddOptions<PowerBiSettings>()
                .Bind(Configuration.GetSection("PowerBI"))
                .ValidateDataAnnotations()
                .Validate(o => o.AreSettingsValid());

            services.AddOptions<AzureActiveDirectorySettings>()
                .Bind(Configuration.GetSection("AzureAd"))
                .ValidateDataAnnotations()
                .Validate(o => o.AreSettingsValid());

            services.AddAuthentication(AzureADDefaults.AuthenticationScheme)
                .AddAzureAD(options => Configuration.Bind("AzureAd", options))
                .AddCookie();

            services.Configure<OpenIdConnectOptions>(AzureADDefaults.OpenIdScheme, options =>
            {
                options.Authority = options.Authority + "/v2.0/";
                options.TokenValidationParameters.ValidateIssuer = false;
            });

            services.AddTransient<Authentication.IAuthenticationHandler, AuthenticationHandler>();
            services.AddTransient<IReportRepository, ReportRepository>();

            services.AddHttpContextAccessor();

            services
                .AddMvc()
                .SetCompatibilityVersion(CompatibilityVersion.Version_2_2)
                .AddRazorPagesOptions(options =>
                {
                    options.Conventions.AuthorizePage("/Reports");
                });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            var builder = new ConfigurationBuilder();

            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();

                builder.AddUserSecrets<Startup>();
            }
            else
            {
                app.UseExceptionHandler("/Error");
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }

            app.UseHttpsRedirection();
            app.UseStaticFiles();

            app.UseAuthentication();

            app.UseMvc();
        }
    }
}

Error output:

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27317.07)
[xUnit.net 00:00:01.30]   Discovering: WebPortal.Int.Tests
[xUnit.net 00:00:01.40]   Discovered:  WebPortal.Int.Tests
[xUnit.net 00:00:01.41]   Starting:    WebPortal.Int.Tests
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]
    User profile is available. Using 'C:\Users\VssAdministrator\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.
info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]
    Creating key {dd820f09-8139-4d7d-954a-399923660f42} with creation date 2019-03-18 22:13:27Z, activation date 2019-03-18 22:13:27Z, and expiration date 2019-06-16 22:13:27Z.
info: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[39]
    Writing data to file 'C:\Users\VssAdministrator\AppData\Local\ASP.NET\DataProtection-Keys\key-dd820f09-8139-4d7d-954a-399923660f42.xml'.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
    Request starting HTTP/2.0 GET http://localhost/Index  
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
    Failed to determine the https port for redirect.
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
    An unhandled exception has occurred while executing the request.
System.ArgumentNullException: Value cannot be null.
Parameter name: uriString
at System.Uri..ctor(String uriString)
at Microsoft.AspNetCore.Authentication.AzureAD.UI.OpenIdConnectOptionsConfiguration.Configure(String name, OpenIdConnectOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass10_0.<Get>b__0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[3]
    An exception was thrown attempting to execute the error handler.
System.ArgumentNullException: Value cannot be null.
Parameter name: uriString
at System.Uri..ctor(String uriString)
at Microsoft.AspNetCore.Authentication.AzureAD.UI.OpenIdConnectOptionsConfiguration.Configure(String name, OpenIdConnectOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass10_0.<Get>b__0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
at System.Lazy`1.CreateValue()
at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
    Request finished in 449.9633ms 500 
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
    Request starting HTTP/2.0 GET http://localhost/Error  
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
    An unhandled exception has occurred while executing the request.
System.ArgumentNullException: Value cannot be null.
Parameter name: uriString
at System.Uri..ctor(String uriString)
at Microsoft.AspNetCore.Authentication.AzureAD.UI.OpenIdConnectOptionsConfiguration.Configure(String name, OpenIdConnectOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass10_0.<Get>b__0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
at System.Lazy`1.CreateValue()
at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[xUnit.net 00:00:02.61]     WebPortal.Int.Tests.AuthenticationTests.Get_PagesNotRequiringAuthenticationWithoutAuthentication_ReturnsSuccessCode(url: "/Index") [FAIL]
[xUnit.net 00:00:02.61]       System.ArgumentNullException : Value cannot be null.
[xUnit.net 00:00:02.61]       Parameter name: uriString
[xUnit.net 00:00:02.61]       Stack Trace:
[xUnit.net 00:00:02.61]            at System.Uri..ctor(String uriString)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Authentication.AzureAD.UI.OpenIdConnectOptionsConfiguration.Configure(String name, OpenIdConnectOptions options)
[xUnit.net 00:00:02.61]            at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
[xUnit.net 00:00:02.61]            at Microsoft.Extensions.Options.OptionsMonitor`1.<>c__DisplayClass10_0.<Get>b__0()
[xUnit.net 00:00:02.61]            at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
[xUnit.net 00:00:02.61]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:00:02.61]            at System.Lazy`1.CreateValue()
[xUnit.net 00:00:02.61]            at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd(String name, Func`1 createOptions)
[xUnit.net 00:00:02.61]            at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.TestHost.HttpContextBuilder.<>c__DisplayClass10_0.<<SendAsync>b__0>d.MoveNext()
[xUnit.net 00:00:02.61]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.TestHost.ClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Mvc.Testing.Handlers.CookieContainerHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[xUnit.net 00:00:02.61]            at Microsoft.AspNetCore.Mvc.Testing.Handlers.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)[xUnit.net 00:00:02.63]     WebPortal.Int.Tests.AuthenticationTests.Get_PagesNotRequiringAuthenticationWithoutAuthentication_ReturnsSuccessCode(url: "/Error") [FAIL]
[xUnit.net 00:00:02.64]     WebPortal.Int.Tests.AuthenticationTests.Get_PagesNotRequiringAuthenticationWithoutAuthentication_ReturnsSuccessCode(url: "") [FAIL]

[xUnit.net 00:00:02.61]            at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
[xUnit.net 00:00:02.61]         D:\a\1\s\WebPortal.Int.Tests\AuthenticationTests.cs(24,0): at WebPortal.Int.Tests.AuthenticationTests.Get_PagesNotRequiringAuthenticationWithoutAuthentication_ReturnsSuccessCode(String url)
[xUnit.net 00:00:02.61]         --- End of stack trace from previous location where exception was thrown ---

edit

It isn't very clear to me where/why I am getting a null reference because as far as I can tell my OpenIdConnectOptions configuration is correct (and it works with AAD SSO).

Answer 1

It turns out that this was because I hadn't called .Build() on my ConfigurationBuilder object that I had created, AND assigned the value to the Configuration field inside the Startup class. This also meant that I moved the code to include the secrets into the Startup constructor.

Even so, my secrets were still not accessible on the build machine (makes sense because they are stored per machine). So I had to add a Command Line task to my build pipeline as well - which uses the dotnet user-secrets set command to add the secrets in that are required for the tests.