How To Prevent HttpWebRequest with long response from timing-out Azure Web App

Question

We are using an HttpWebRequest in C# to get data from an internet resource in our Azure Web App. The problem is that Azure has a limitation on how long it keeps the connection alive (around 240 seconds). Due to the nature of our application, the response will sometimes take longer than 240 seconds. When this happens, the webpage will go white, and the "View Source" will show zero source code (which has made this issue difficult to debug).

Here's some sample code to illustrate:

        webRequest = WebRequest.Create(PAGE_URL) as HttpWebRequest;
        webRequest.Method = "POST";
        webRequest.ContentType = "application/x-www-form-urlencoded";
        webRequest.CookieContainer = cookies;
        webRequest.Timeout = Timeout.Infinite;
        webRequest.KeepAlive = true;
        StreamWriter requestWriter2 = new
            StreamWriter(webRequest.GetRequestStream());
        requestWriter2.Write(postString); 
        requestWriter2.Close();
        WebResponse response = webRequest.GetResponse();
        Stream stream = response.GetResponseStream();

Adding webRequest.Timeout and webRequest.KeepAlive did not solve the issue.

jbq on this thread mentioned he had a workaround by sending a "newline character every 5 seconds", however did not explain how to accomplish this exactly. He was answering a question about an Azure VM, but I think an Azure Web App would have similar behaviors with respect to what I believe are the load balancers that are responsible for the timeout.

The Question: How can I send one HttpWebRequest, and then send another HttpWebRequest while the previous one is running with a blank line to maintain the connection and prevent the Azure load balancer(?) from timing out the azure application? Would a new session variable need to be used? Perhaps an asynchronous method? Do I need to send the "pinging" request before the main request? If so, how would this look in implementation? Or is it something else entirely? Please provide some source code as an example :)

Note: you do not need to use an HttpWebRequest to replicate this issue. Attach a debugger from within Visual Studio to a live Azure Web App. Place a breakpoint within Visual Studio at any piece of code. When that breakpoint is hit, after roughly 4 minutes you'll see the page in your browser stop loading and go white with an empty source. So, it's not specifically related to HttpWebRequest, but that is an operation that would typically cause this sort of issue since some responses take longer than 4 minutes.

*EDIT: I think what I am looking for is an implementation of Asynchronous methods. I will update this post as I find a satisfactory implementation.

Answer 1

If you are making an HttpWebRequest to an Azure Website then you use ServicePointManager.SetTcpKeepAlive on your client code which uses HttpWebRequest.

https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.settcpkeepalive(v=vs.110).aspx

The 4 minute timeout that you are talking about is an IDLE timeout over the TCP layer and setting this will ensure that your client (who is using HttpWebRequest) sends ACK packet over TCP so that the connection doesn't get idle.

If your web application is making a HttpWebRequest to some other service, you can still use this function but that will just ensure that the Idle timeout is not hit when calling the remote service. Your actual HTTP request to the Azure Webapp may still hit the 4 minute time and if the client to your Azure web app is not HttpWebRequest, then again the 4 minute idle timeout will bite you...

The best thing to do here is to change the code a bit to implement a JOB Model kind of pattern where-in you make a server call which returns a JOBID. The client then queries the server using this JOBID in a polling fashion and when the job completes on the server the status of this JOBID can be set to COMPLETED in which case the client can then retrieve the data. You can use Webjobs in Azure Webapps to achieve something like this.

Hope this helps...