HttpWebRequest with caching enabled throws exceptions

Question

I'm working on a small C#/WPF application that interfaces with a web service implemented in Ruby on Rails, using handcrafted HttpWebRequest calls and JSON serialization. Without caching, everything works as it's supposed to, and I've got HTTP authentication and compression working as well.

Once I enable caching, by setting request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);, things go awry - in the production environment. When connecting to a simple WEBrick instance, things work fine, I get HTTP/1.1 304 Not Modified as expected and HttpWebRequest delivers the cached content.

When I try the same against the production server, running nginx/0.8.53 + Phusion Passenger 3.0.0, the application breaks. First request (uncached) is served properly, but on the second request which results in the 304 response, I get a WebException stating that "The request was aborted: The request was canceled." as soon as I invoke request.GetResponse().

I've run the connections through fiddler, which hasn't helped a whole lot; both WEBrick and nginx return an empty entity body, albeit different response headers. Intercepting the request and changing the response headers for nginx to match those of WEBrick didn't change anything, leading me to think that it could be a keep-alive issue; setting request.KeepAlive = false; changes nothing, though - it doesn't break stuff when connecting to WEBrick, and it doesn't fix stuff when connecting to nginx.

For what it's worth, the WebException.InnerException is a NullReferenceException with the following StackTrace:

at System.Net.HttpWebRequest.CheckCacheUpdateOnResponse()
at System.Net.HttpWebRequest.CheckResubmitForCache(Exception& e)
at System.Net.HttpWebRequest.DoSubmitRequestProcessing(Exception& exception)
at System.Net.HttpWebRequest.ProcessResponse()
at System.Net.HttpWebRequest.SetResponse(CoreResponseData coreResponseData)

Headers for the (working) WEBrick connection:

########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testbox.local:3030
If-None-Match: "84a49062768e4ca619b1c081736da20f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
X-Ua-Compatible: IE=Edge
Etag: "84a49062768e4ca619b1c081736da20f"
Date: Wed, 01 Dec 2010 18:18:59 GMT
Server: WEBrick/1.3.1 (Ruby/1.8.7/2010-08-16)
X-Runtime: 0.177545
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: *REDACTED*

Headers for the (exception-throwing) nginx connection:

########## request
GET /users/current.json HTTP/1.1
Authorization: Basic *REDACTED*
Content-Type: application/json
Accept: application/json
Accept-Charset: utf-8
Host: testsystem.local:8080
If-None-Match: "a64560553465e0270cc0a23cc4c33f9f"
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
########## response
HTTP/1.1 304 Not Modified
Connection: keep-alive
Status: 304
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.0
ETag: "a64560553465e0270cc0a23cc4c33f9f"
X-UA-Compatible: IE=Edge,chrome=1
X-Runtime: 0.240160
Set-Cookie: *REDACTED*
Cache-Control: max-age=0, private, must-revalidate
Server: nginx/0.8.53 + Phusion Passenger 3.0.0 (mod_rails/mod_rack)

UPDATE:

I tried doing a quick-and-dirty manual ETag cache, but turns out that's a no-go: I get a WebException when invoking request.GetResponce(), telling me that "The remote server returned an error: (304) Not Modified." - yeah, .NET, I kinda knew that, and I'd like to (attempt to) handle it myself, grr.

UPDATE 2:

Getting closer to the root of the problem. The showstopper seems to be a difference in the response headers for the initial request. WEBrick includes a Date: Wed, 01 Dec 2010 21:30:01 GMT header, which isn't present in the nginx reply. There's other differences as well, but intercepting the initial nginx reply with fiddler and adding a Date header, the subsequent HttpWebRequests are able to process the (unmodified) nginx 304 replies.

Going to try to look for a workaround, as well as getting nginx to add the Date header.

UPDATE 3:

It seems that the serverside issue is with Phusion Passenger, they have an open issue about lack of the Date header. I'd still say that HttpWebRequest's behavior is... suboptimal.

UPDATE 4:

Added a Microsoft Connect ticket for the bug.

Answer 1

So, it turns out to be Phusion Passenger (or nginx, depending on how you look at it - and Thin as well) that doesn't add a Date HTTP response header, combined with what I see as a bug in .NET HttpWebRequest (in my situation there's no If-Modified-Since, thus Date shouldn't be necessary) leading to the problem.

The workaround for this particular case was to edit our Rails ApplicationController:

class ApplicationController < ActionController::Base
    # ...other stuff here

    before_filter :add_date_header
    # bugfix for .NET HttpWebRequst 304-handling bug and various
    # webservers' lazyness in not adding the Date: response header.
    def add_date_header
        response.headers['Date'] = Time.now.to_s
    end
end

UPDATE:

Turns out it's a bit more complex than "just" setting HttpRequestCachePolicy - to repro, I also need to have manually constructed HTTP Basic Auth. So the involved components are the following:

1. HTTP server that doesn't include a HTTP "Date:" response header.
2. manual construction of HTTP Authorization request header.
3. use of HttpRequestCachePolicy.

Smallest repro I've been able to come up with:

namespace Repro
{
    using System;
    using System.IO;
    using System.Net;
    using System.Net.Cache;
    using System.Text;

    class ReproProg
    {
        const string requestUrl = "http://drivelog.miracle.local:3030/users/current.json";

        // Manual construction of HTTP basic auth so we don't get an unnecessary server
        // roundtrip telling us to auth, which is what we get if we simply use
        // HttpWebRequest.Credentials.
        private static void SetAuthorization(HttpWebRequest request, string _username, string _password)
        {
            string userAndPass = string.Format("{0}:{1}", _username, _password);
            byte[] authBytes = Encoding.UTF8.GetBytes(userAndPass.ToCharArray());
            request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(authBytes);
        }

        static public void DoRequest()
        {
            var request = (HttpWebRequest) WebRequest.Create(requestUrl);

            request.Method = "GET";
            request.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);
            SetAuthorization(request, "[email protected]", "12345678");

            using(var response = request.GetResponse())
            using(var stream = response.GetResponseStream())
            using(var reader = new StreamReader(stream))
            {
                string reply = reader.ReadToEnd();
                Console.WriteLine("########## Server reply: {0}", reply);
            }
        }

        static public void Main(string[] args)
        {
            DoRequest();    // works
            DoRequest();    // explodes
        }
    }
}

Answer 2

I think the designers find it reasonable to throw an exception when the "expected behavior"---i.e., getting a response body---cannot be completed. You can handle this somewhat intelligently as follows:

catch (WebException ex)
{
    if (ex.Status == WebExceptionStatus.ProtocolError)
    {
        var statusCode = ((HttpWebResponse)ex.Response).StatusCode;
        // Test against HttpStatusCode enumeration.
    }
    else
    {
        // Do something else, e.g. throw;
    }
}