OAuthBearerAuthenticationMiddleware - Server cannot append header after HTTP headers have been sent

Question

I've been trying to plug some OWIN middleware into an existing WebApi project. My start-up originally contained only the following lines:

application.UseOAuthBearerAuthentication(newOAuthBearerAuthenticationOptions());
application.UseWebApi(config);

With this configuration, I was intermittently, and mainly after an iisreset, receiving malformed responses (as identified by fiddler) that were being caused by the middleware trying to add headers but after the response had been sent, this was being reported as the exception:

Server cannot append header after HTTP headers have been sent.

I recompiled Microsoft.Owin.Security.OAuth to add some additional tracing to show the order that thing were happening and I got the following output:

Microsoft.Owin.Security.OAuth.OAuthBearerAuthenticationMiddleware Information: 0 : : OAUTH: Authenticating...
System.Web.Http.Request: ;;http://localhost:555/entityinstance/member    
System.Web.Http.Controllers: WebHostHttpControllerTypeResolver;GetControllerTypes;: 
System.Web.Http.Controllers: WebHostHttpControllerTypeResolver;GetControllerTypes;: 
System.Web.Http.MessageHandlers: LogHandler;SendAsync;: 
System.Web.Http.Controllers: DefaultHttpControllerSelector;SelectController;Route='entityDefinitionName:member,controller:EntityInstance'
System.Web.Http.Controllers: DefaultHttpControllerSelector;SelectController;EntityInstance
System.Web.Http.Controllers: HttpControllerDescriptor;CreateController;: 
System.Web.Http.Controllers: WindsorCompositionRoot;Create;: 
System.Web.Http.Controllers: WindsorCompositionRoot;Create;Loyalty.Services.WebApi.Controllers.EntityInstanceController
System.Web.Http.Controllers: HttpControllerDescriptor;CreateController;Loyalty.Services.WebApi.Controllers.EntityInstanceController
System.Web.Http.Controllers: EntityInstanceController;ExecuteAsync;: 
System.Web.Http.Action: ApiControllerActionSelector;SelectAction;: 
System.Web.Http.Action: ApiControllerActionSelector;SelectAction;Selected action 'Get(String entityDefinitionName)'
System.Net.Http.Formatting: DefaultContentNegotiator;Negotiate;Type='HttpError', formatters=[JsonMediaTypeFormatterTracer, XmlMediaTypeFormatterTracer, FormUrlEncodedMediaTypeFormatterTracer, FormUrlEncodedMediaTypeFormatterTracer]
System.Net.Http.Formatting: JsonMediaTypeFormatter;GetPerRequestFormatterInstance;Obtaining formatter of type 'JsonMediaTypeFormatter' for type='HttpError', mediaType='application/json; charset=utf-8'
System.Net.Http.Formatting: JsonMediaTypeFormatter;GetPerRequestFormatterInstance;Will use same 'JsonMediaTypeFormatter' formatter
System.Net.Http.Formatting: DefaultContentNegotiator;Negotiate;Selected formatter='JsonMediaTypeFormatter', content-type='application/json; charset=utf-8'
System.Web.Http.Controllers: EntityInstanceController;ExecuteAsync;: 
System.Net.Http.Formatting: JsonMediaTypeFormatter;WriteToStreamAsync;Value='System.Web.Http.HttpError', type='HttpError', content-type='application/json; charset=utf-8'
System.Net.Http.Formatting: JsonMediaTypeFormatter;WriteToStreamAsync;: 
System.Web.Http.Request: ;;Content-type='application/json; charset=utf-8', content-length=68
System.Web.Http.Controllers: EntityInstanceController;Dispose;: 
System.Web.Http.Controllers: EntityInstanceController;Dispose;: 
Microsoft.Owin.Security.OAuth.OAuthBearerAuthenticationMiddleware Information: 0 : : OAUTH: Applying Challange...
A first chance exception of type 'System.Web.HttpException' occurred in System.Web.dll

So what it looks like is that the response handling part of the middleware, following the Russian doll model is trying to modify headers, but the response has already been completed at a previous stage. I have tried adding different stage markers to control this behavior, but nothing seems to help.

The surprising thing after seeing that trace, was that it wasn't happening all the time. I wrote my own version of this middleware with a more minimal implementation and after registering this, in place of the MS one, I did indeed start to see the error on every request, I'm wondering if it was always being thrown, but swallowed on occasion or whether fiddler wasn't waiting around for long enough to see it.

My current best guess is this problem is happening as a result of using a different HttpConfiguration for Owin than the WebApi setup. Unfortnately, I can't swap out the entirety of the HTTPApplication and go over to OWIN lock stock because of way that delegating handlers run inside a different context in OWIN, where you don't have access to route data, as this would break a lot of our existing infrastructure.

Can any give me any pointers as to what's going on here, is this a supported scenario? Am I missing something obvious?

Answer 1

Ok, well, I've solved it!

Short Answer

You get this problem if you use two HttpConfiguration instances. You must ensure that you use the same HttpConfiguration for both your call to application.UseWebApi(config); and for your webapi config.

I'm guessing that the reason for this is that unless you use the same config, then the runtime as no way of knowing when the response is ready to be sent as there isn't a single place it can look to determine if all your handlers have run.

Medium Sized Answer

When converting an existing webapi application, you'll generally have your container bootstrap and your web api configuration registration in your global.asax application_start handler. When moving to OWIN, one of the first things you'll do is add a Startup class which you use to configure your OWIN application via the appbuilder. In this scenario, it's quite easy to follow the pure OWIN examples and new up a new HttpConfiguration in your startup whilst leaving your existing registration utilising GlobalConfiguration. You'll end up with something like:

Global.asax:

    protected void Application_Start()
    {
        var config =  GlobalConfiguration.Configuration;
        Bootstrapper.Run(config);            
        WebApiConfig.Register(config);
     }

Startup.cs:

  public void Configuration(IAppBuilder application)
            {
                var config = new HttpConfiguration();
                application.UseOAuthAuthorizationServer(new OAuthAuthorizationServerOptions());
                application.UseOAuthBearerAuthentication(new OAuthBearerAuthenticationOptions());

                config.MapHttpAttributeRoutes();
                application.UseWebApi(config);   
            }

When what you really need is:

public void Configuration(IAppBuilder application)
        {
            var config = new HttpConfiguration();

            Bootstrapper.Run(config);

            application.UseOAuthAuthorizationServer(new OAuthAuthorizationServerOptions());
            application.UseOAuthBearerAuthentication(new OAuthBearerAuthenticationOptions());

            WebApiConfig.Register(config);
            config.MapHttpAttributeRoutes();
            application.UseWebApi(config);   
        }

Longer Answer

You might be thinking that this is fairly obvious, and that if you followed the examples closely you'd figure this out pretty quickly, and you'd be right. This is something that I tried shortly after coming to the problem (I didn't write the original code ;)). Unfortunately, if you try the above with some standard web.configs used in WebApi projects, you'll run into a number of other issues which present problems which you think might be related to your original issue, but aren't.

Problem: 404 on each request.

Solution: You need the following handler registered:

 <!-- language: lang-xml -->
  <handlers accessPolicy="Read, Execute, Script">
    <remove name="WebDAV" />    
    <remove name="ExtensionlessUrlHandler-Integrated-4.0" />      
    <remove name="TRACEVerbHandler" />
    <add name="ExtensionlessUrlHandler-Integrated-4.0" path="*" verb="*" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
</handlers>

Problem: Unable to redirect after the response has been sent / 404 redirect to login.aspx after you issue a 401.

Solution: You need to unregister the formsAuthentication module:

<!-- language: lang-xml -->
<modules runAllManagedModulesForAllRequests="true">
      <remove name="FormsAuthentication" />
</modules>

Problem: "message": "No HTTP resource was found that matches the request URI 'http://blah'.", "messageDetail": "No type was found that matches the controller named 'blah'."

Solution: This is a subtle one. We were using a delegating handler to scan our controller actions for an Authenticate attribute. We were doing this with the following code:

public virtual T ScanForAttribute<T>(HttpRequestMessage request, HttpConfiguration config) where T : Attribute
        {
            var controllerSelector = new DefaultHttpControllerSelector(config);
            var descriptor = controllerSelector.SelectController(request);

            .. some other stuff
        }

Now, the problem we have under OWIN is that controllerSelector.SelectController (implemented in System.Web.Http) internally relies on the MS_RequestContext request property, if it doesn't find this then it throws an HttpResponseException with a status code of 404, which results in a 404 response being sent, hence the above issue. You can get this working in OWIN with a bit of hack:

public virtual T ScanForAttribute<T>(HttpRequestMessage request, HttpConfiguration config) where T : Attribute
        {
            var data = request.GetConfiguration().Routes.GetRouteData(request);
            ((HttpRequestContext) request.Properties["MS_RequestContext"]).RouteData = data;

            var controllerSelector = new DefaultHttpControllerSelector(config);
            var descriptor = controllerSelector.SelectController(request);
            .. Some other stuff
        }

Problem: You try to resolve the callers IP using:

if (request.Properties.ContainsKey("MS_HttpContext"))
                {
                    return ((HttpContextWrapper)request.Properties["MS_HttpContext"]).Request.UserHostAddress;
                }

Solution: You need to also check the following, for it to work under OWIN:

  if (request.Properties.ContainsKey("MS_OwinContext"))
                {
                    OwinContext owinContext = (OwinContext)request.Properties["MS_OwinContext"];
                    if (owinContext != null)
                    {
                        return owinContext.Request.RemoteIpAddress;
                    }
                } 

And.... we're now working well! Troublshooting this issue would have been much easier if there was a little more trace output from OWIN, but unfortunately a lot of the middleware in the katana project is a little bit quiet on the trace front, hopefully this is something that will be addressed over time!

Hope this helps.