IIS7 Application Request Routing (arr reverse proxy) combined with managed module - time out

Question

I am trying to build a proxy that would serve requests to an internal site (hiding the origin) but at the same time inspect the packets and asynchronously post-process them.

E.g. let's say all SOAP calls to http://www.foo.com will go to http://192.168.1.1, and at the same time be stored in a DB for post analysis. The internal server is a black box, so changing something on it is out of this question scope.

Anyway, I have configured ARR, with reverse proxy, made URL rewrite filter with wildcards, all works flawless. Then, I tried to add an managed HttpModule written in C#, and hooked to Application_BeginRequest and Application_EndRequest. I am able to access request headers, response headers on end request (app pool being in integrated mode) and even able to read response content from the outputstream by setting a filter on Response.Filter, that caches all writes in an additional memory stream.

The problem is that the moment I try to read (inside the module BeginRequest handler) the input stream from the request, ARR stays a while and throws a

HTTP Error 502.3 - Bad Gateway The operation timed out Handler ApplicationRequestRoutingHandler Error Code 0x80072ee2

So it times out.

Looking with Failed Request Tracing I see:

MODULE_SET_RESPONSE_ERROR_STATUS Warning ModuleName="ApplicationRequestRouting", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="502", HttpReason="Bad Gateway", HttpSubStatus="3", ErrorCode="2147954402", ConfigExceptionInfo="" SET_RESPONSE_ERROR_DESCRIPTION Warning ErrorDescription="The operation timed out"

Now any similar posts on the net didn't helped as this isn't a timeout error (proxy has 120 seconds setting, page answers in under 100 ms), and the moment I comment the code of the handler that tries to read FORM data or InputStream data, everything works as a charm.

Even if I set the position of the inputstream to 0 after reading it, I still get timeouts. If I read the input stream on EndRequest, it gets 0 bytes, even if it was a POST request. (which is clearly wrong)

Does ARR has a bug in the fact that I try to read an input stream before it tries to re-route it?

Things used: Windows Server 2008 R2 IIS 7.5 ARR v2 .Net Framework 3.5 module

Ideas? Thanks /Cosmin

Answer 1

If you can switch to .Net Framework 4, there is a solution for this.

After you are done with your BeginRequest/EndRequest in your HttpModule event handler, add a call to HttpRequest.InsertEntityBody.

    /* BeginRequest event: Executes before request is processed */
    private void Application_BeginRequest(Object source, EventArgs e)
    {
        HttpApplication application = (HttpApplication)source;
        HttpRequest request = application.Context.Request;

        // Do something with request
        DoMyOwnRequestProcessing(request);

        // After you finish, make sure IIS gets the entity body
        // For example, Application Request Routing needs this
        request.InsertEntityBody();
    }

Take a look at this on MSDN: HttpRequest.InsertEntityBody.

Answer 2

I know this is a year old question, but I just went through the same thing and found a solution. So, I'm posting it here for anyone else that runs into this.

In my case I only saw the timeout issue with POST requests.

It appears that the 2.0/2.1 ARR assumes that the input stream will be at the start of the posted data. However, the following code (for example) will break this assumption:

    HttpContext context = HttpContext.Current;
    HttpRequest request = context.Request;

    string value = request.Params["Name"];

The key is how Params is described

    Gets a combined collection of System.Web.HttpRequest.QueryString,
    System.Web.HttpRequest.Form, System.Web.HttpRequest.ServerVariables,
    and System.Web.HttpRequest.Cookies items."`

When the request is a POST, accessing Params will read the posted data from the input stream, invalidating ARR's assumption. As will reading from the input stream.

I knew the data I needed was in the query string, not the posted data, so I worked around this by accessing the QueryString instead of Params. This avoids reading the posted data and works fine for me.

    string value = request.QueryString["Name"];

This issue appears to be fixed in ARR 2.5.

Upgrading ARR appears to be the only solution if you need to access posted data before handing off to ARR. The key is to let HttpRequest handle acquiring the data into Params. If you read it directly it will not work.

Answer 3

I just ran into this bug and your experiences helped me determine the root cause.

My main server is MVC based and it looks at the Request.Form values in the Application_BeginRequest method. If the form values are accessed ARR fails to forward the body of a HTTP POST request. GET requests will work fine since there is no body.

I have routes.IgnoreRoute ("Forum/{*pathInfo}"); as a registered route but ARR runs as a HttpModule and doesn't kick-in until later in the pipeline. That means my MVC based application is given the opportunity to access the content of the POST body which somehow prevents ARR from accessing the body itself and forwarding it to the proxy'd server.

Here is Cosmin's related post on the iis.net forums: ARR 2.0 BUG - combined with managed http module timeout on read inputstream

In my application I have all myserver.com/Forum/* requests being reverse proxy'd to a separate application on another server. So I simply checked the HttpContext.Current.Request.Url in my MVC application's Application_BeginRequest method to make sure it does not contain /Forum before accessing the Request.Form values. Once I did that the POST bodies made it through ARR just fine.

UPDATE: after further testing it appears that there are still problems with ARR as POST from non-authenticated users still fails. Instead of the main website being MVC I created a dummy IIS .NET 4.0 website with a single Default.html document. But I still ran into problems with POST requests and ARR. Then I switch the application pool to ASP.NET 2.0 and what do you know, it works. At this point I have to assume that something in the .NET 4.0 pipeline is accessing the input stream which prevents ARR from accessing the input stream itself in order to forward the POST body.