How to get the HTTP response when the request stream was closed during transfer

Question

TL;DR version

When a transfer error occurs while writing to the request stream, I can't access the response, even though the server sends it.

---

Full version

I have a .NET application that uploads files to a Tomcat server, using HttpWebRequest. In some cases, the server closes the request stream prematurely (because it refuses the file for one reason or another, e.g. an invalid filename), and sends a 400 response with a custom header to indicate the cause of the error.

The problem is that if the uploaded file is large, the request stream is closed before I finish writing the request body, and I get an IOException:

Message: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
InnerException: SocketException: An existing connection was forcibly closed by the remote host

I can catch this exception, but then, when I call GetResponse, I get a WebException with the previous IOException as its inner exception, and a null Response property. So I can never get the response, even though the server sends it (checked with WireShark).

Since I can't get the response, I don't know what the actual problem is. From my application point of view, it looks like the connection was interrupted, so I treat it as a network-related error and retry the upload... which, of course, fails again.

How can I work around this issue and retrieve the actual response from the server? Is it even possible? To me, the current behavior looks like a bug in HttpWebRequest, or at least a severe design issue...

---

Here's the code I used to reproduce the problem:

var request = HttpWebRequest.CreateHttp(uri);
request.Method = "POST";
string filename = "foo\u00A0bar.dat"; // Invalid characters in filename, the server will refuse it
request.Headers["Content-Disposition"] = string.Format("attachment; filename*=utf-8''{0}", Uri.EscapeDataString(filename));
request.AllowWriteStreamBuffering = false;
request.ContentType = "application/octet-stream";
request.ContentLength = 100 * 1024 * 1024;

// Upload the "file" (just random data in this case)
try
{
    using (var stream = request.GetRequestStream())
    {
        byte[] buffer = new byte[1024 * 1024];
        new Random().NextBytes(buffer);
        for (int i = 0; i < 100; i++)
        {
            stream.Write(buffer, 0, buffer.Length);
        }
    }
}
catch(Exception ex)
{
    // here I get an IOException; InnerException is a SocketException
    Console.WriteLine("Error writing to stream: {0}", ex);
}

// Now try to read the response
try
{
    using (var response = (HttpWebResponse)request.GetResponse())
    {
        Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
    }
}
catch(Exception ex)
{
    // here I get a WebException; InnerException is the IOException from the previous catch
    Console.WriteLine("Error getting the response: {0}", ex);
    var webEx = ex as WebException;
    if (webEx != null)
    {
        Console.WriteLine(webEx.Status); // SendFailure
        var response = (HttpWebResponse)webEx.Response;
        if (response != null)
        {
            Console.WriteLine("{0} - {1}", (int)response.StatusCode, response.StatusDescription);
        }
        else
        {
            Console.WriteLine("No response");
        }
    }
}

---

Additional notes:

If I correctly understand the role of the 100 Continue status, the server shouldn't send it to me if it's going to refuse the file. However, it seems that this status is controlled directly by Tomcat, and can't be controlled by the application. Ideally, I'd like the server not to send me 100 Continue in this case, but according to my colleagues in charge of the back-end, there is no easy way to do it. So I'm looking for a client-side solution for now; but if you happen to know how to solve the problem on the server side, it would also be appreciated.

The app in which I encounter the issue targets .NET 4.0, but I also reproduced it with 4.5.

I'm not timing out. The exception is thrown long before the timeout.

I tried an async request. It doesn't change anything.

I tried setting the request protocol version to HTTP 1.0, with the same result.

---

Someone else has already filed a bug on Connect for this issue: https://connect.microsoft.com/VisualStudio/feedback/details/779622/unable-to-get-servers-error-response-when-uploading-file-with-httpwebrequest

Answer 1

I am out of ideas as to what can be a client side solution to your problem. But I still think the server side solution of using a custom tomcat valve can help here. I currently doesn`t have a tomcat setup where I can test this but I think a server side solution here would be along the following lines :

RFC section 8.2.3 clearly states : Requirements for HTTP/1.1 origin servers:

  - Upon receiving a request which includes an Expect request-header
    field with the "100-continue" expectation, an origin server MUST
    either respond with 100 (Continue) status and continue to read
    from the input stream, or respond with a final status code. The
    origin server MUST NOT wait for the request body before sending
    the 100 (Continue) response. If it responds with a final status
    code, it MAY close the transport connection or it MAY continue
    to read and discard the rest of the request.  It MUST NOT
    perform the requested method if it returns a final status code.

So assuming tomcat confirms to the RFC, while in the custom valve you would have recieved the HTTP request header, but the request body would not be sent since the control is not yet in the servlet that reads the body.

So you can probably implement a custom valve, something similar to :

import org.apache.catalina.connector.Request;
import org.apache.catalina.connector.Response;
import org.apache.catalina.valves.ErrorReportValve;

public class CustomUploadHandlerValve extends ValveBase {

    @Override
    public void invoke(Request request, Response response) throws IOException, ServletException {
         HttpServletRequest httpRequest = (HttpServletRequest) request;
         String fileName = httpRequest.getHeader("Filename");  // get the filename or whatever other parameters required as per your code
         bool validationSuccess = Validate(); // perform filename check or anyother validation here
         if(!validationSuccess)
         {
             response = CreateResponse(); //create your custom 400 response here
             request.SetResponse(response);
             // return the response here
         }
         else
         {
             getNext().invoke(request, response); // to pass to the next valve/ servlet in the chain
         }
    }
    ...
}

DISCLAIMER : Again I haven`t tried this to success, need sometime and a tomcat setup to try it out ;). Thought it might be a starting point for you.

Answer 2

I had the same problem. The server sends a response before the client end of the transmission of the request body, when I try to do async request. After a series of experiments, I found a workaround. After the request stream has been received, I use reflection to check the private field _CoreResponse of the HttpWebRequest. If it is an object of class CoreResponseData, I take his private fields (using reflection): m_StatusCode, m_StatusDescription, m_ResponseHeaders, m_ContentLength. They contain information about the server's response! In most cases, this hack works!

Answer 3

What are you getting in the status code and response of the second exception not the internal exception?

If a WebException is thrown, use the Response and Status properties of the exception to determine the response from the server.

http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse(v=vs.110).aspx