Web API as a Proxy and Chunked Transfer Encoding

Question

I have been playing around with using Web API (Web Host) as a proxy server and have run into an issue with how my Web API proxy handles responses with the "Transfer-Encoding: chunked" header.

When bypassing the proxy, the remote resource sends the following response headers:

Cache-Control:no-cache
Content-Encoding:gzip
Content-Type:text/html
Date:Fri, 24 May 2013 12:42:27 GMT
Expires:-1
Pragma:no-cache
Server:Microsoft-IIS/8.0
Transfer-Encoding:chunked
Vary:Accept-Encoding
X-AspNet-Version:4.0.30319
X-Powered-By:ASP.NET

When going through my Web API based proxy, my request hangs unless I explicitly reset the TransferEncodingChunked property on the response header to false:

response.Headers.TransferEncodingChunked = false;

I admit, I don't fully understand what impact setting the TransferEncodingChunked property has, but it seems strange to me that in order to make the proxy work as expected, I need to set this property to false when clearly the incoming response has a "Transfer-Encoding: chunked" header. I am also concerned about side effects to explicitly setting this property. Can anyone help me understand what is going on and why setting this property is required?

UPDATE: So I did a little more digging into the difference in the response when going through the proxy vs. not. Whether I explicitly set the TransferEncodingChunked property to false, the response headers when coming through the proxy are exactly the same as when not going through the proxy. However, the response content is different. Here are a few samples (I turned off gzip encoding):

// With TransferEncodingChunked = false
2d\r\n
This was sent with transfer-encoding: chunked\r\n
0\r\n

// Without explicitly setting TransferEncodingChunked
This was sent with transfer-encoding: chunked

Clearly, the content sent with TransferEncodingChunked set to false is in fact transfer encoded. This is actually the correct response as it is what was received from the requested resource behind the proxy. What continues to be strange is the second scenario in which I don't explicitly set TransferEncodingChunked on the response (but it is in the response header received from the proxied service). Clearly, in this case, the response is NOT in fact transfer encoded by IIS, in spite of the fact that the actual response is. Strange...this is starting to feel like designed behavior (in which case, I'd love to know how / why) or a bug in IIS, ASP.Net, or Web API.

Here is a simplified version of the code I am running:

Proxy Web API application:

// WebApiConfig.cs
config.Routes.MapHttpRoute(
    name: "Proxy",
    routeTemplate: "{*path}",
    handler: HttpClientFactory.CreatePipeline(
        innerHandler: new HttpClientHandler(), // Routes the request to an external resource
        handlers: new DelegatingHandler[] { new ProxyHandler() }
    ),
    defaults: new { path = RouteParameter.Optional },
    constraints: null
);

// ProxyHandler.cs
public class ProxyHandler : DelegatingHandler
{
    protected override async System.Threading.Tasks.Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, System.Threading.CancellationToken cancellationToken)
    {
        // Route the request to my web application
        var uri = new Uri("http://localhost:49591" + request.RequestUri.PathAndQuery);
        request.RequestUri = uri;

        // For GET requests, somewhere upstream, Web API creates an empty stream for the request.Content property
        // HttpClientHandler doesn't like this for GET requests, so set it back to null before sending along the request
        if (request.Method == HttpMethod.Get)
        {
            request.Content = null;
        }

        var response = await base.SendAsync(request, cancellationToken);

        // If I comment this out, any response that already has the Transfer-Encoding: chunked header will hang in the browser
        response.Headers.TransferEncodingChunked = false;

        return response;
    }
}

And my web application controller which creates a "chunked" response (also Web API):

public class ChunkedController : ApiController
{
    public HttpResponseMessage Get()
    {
        var response = Request.CreateResponse(HttpStatusCode.OK);

        var content = "This was sent with transfer-encoding: chunked";
        var bytes = System.Text.Encoding.ASCII.GetBytes(content);
        var stream = new MemoryStream(bytes);

        response.Content = new ChunkedStreamContent(stream);

        return response;
    }
}

public class ChunkedStreamContent : StreamContent
{
    public ChunkedStreamContent(Stream stream)
        : base(stream) { }

    protected override bool TryComputeLength(out long length)
    {
        length = 0L;
        return false;
    }
}

Answer 1

From an HttpClient standpoint, content chunking is essentially a detail of the transport. The content provided by response.Content is always de-chunked for you by HttpClient.

It looks like there's a bug in Web API that it doesn't correctly (re-)chunk content when requested by the response.Headers.TransferEncodingChunked property when running on IIS. So the problem is that the proxy is telling the client, via the headers, that the content is chunked when in fact it is not. I've filed the bug here: https://aspnetwebstack.codeplex.com/workitem/1124

I think your workaround is the best option at the moment.

Also notice that you have multiple layers here that likely weren't designed/tested for proxying scenarios (and may not support it). On the HttpClient side, note that it will automatically decompress and follow redirects unless you turn that behavior off. At a minimum, you'll want to set these two properties: http://msdn.microsoft.com/en-us/library/system.net.http.httpclienthandler.allowautoredirect.aspx http://msdn.microsoft.com/en-us/library/system.net.http.httpclienthandler.automaticdecompression.aspx

On the WebApi/IIS side, you've found at least one bug, and it wouldn't be suprising to find others as well. Just be forewarned there may be bugs like this currently writing a proxy using these technologies outside their main design use cases.