Why HttpWebRequest.GetRequestStream() tries to connect

Question

Maybe this seems like weird question, but I came across the following situation:

I try to make a post request to a service, and to add the post-data I chose make a Stream out of the request and use a StreamWriter to write the body on it.

But, before I actually execute the request (with GetResponse), even before I write to the stream object, I get an "Unable to connect exception" exactly on

var stream = request.GetRequestStream();

After a little investigation, I realized that request.GetRequestStream() is actually trying to connect. The problem in my case was network connectivity to the server (firewall issue).

BUT my question here is Why HttpWebRequest.GetRequestStream() tries to connect???

My simple thought was that, while on the request creation, there is no connection to the server of the request.

I found some related questions, such like this

But it does not seem to ansewr my question exactly.

Any explanation please?

PS: Any suggestion of how to avoid this "early" connection effect would be much appreciated.

Answer 1

.NET I/O APIs generally operate on streams, which are APIs that allow developers to read and write an ordered sequence of data. By making reading and writing into generic APIs, it enables generic libraries to operate on streams to do powerful things: compression, encryption, encoding, etc. (BTW, treating different kinds of I/O similarly has a long history, most famously in UNIX where everything is a file.)

Although reading and writing data works pretty similarly across many different kinds of streams, opening a stream is much harder to make generic. Think about the vastly different APIs you use to open a file vs. make an HTTP request vs. execute a database query.

Therefore, .NET's Stream class has no generic Open() method because getting a stream into an opened state is very different between different types of streams. Instead, the streams APIs expect to be given a stream that's already open, where "open" means that it's ready to be written to and/or read from.

Therefore, in .NET there's a typical pattern for I/O:

1. Write some resource-specific code to open a stream. These APIs generally return an open stream.
2. Hand off that open stream to generic APIs that read and/or write from it.
3. Close the stream (also generic) when you're done.

Now think about how that pattern above aligns to an HTTP request, which has the following steps:

• a. Lookup the server's IP address in DNS
• b. Make a TCP connection to the server
• c. Send the URL and request headers to the server
• d. If it's a POST (or PUT or other method that sends a request body) then upload the request body. If it's a GET, this is a no-op.
• e. Now read the response
• f. Finally, close the connection.

(I'm ignoring a lot of real-world complexity in the steps above like SSL, keep-alive connections, cached responses, etc. but the basic workflow is accurate enough to answer your question.)

OK now put yourself in the shoes of the .NET team trying to build an HTTP client API, remembering to split the non-generic parts ("get an open stream") from the generic parts: read and/or write, and then close the stream.

If your API only had to handle GET requests, then you'd probably make the connection while executing the same API that returns the response stream. This is exactly what HttpWebRequest.GetResponse does.

But if you're sending POST requests (or PUT or other similar methods), then you have to upload data to the server. Unlike HTTP headers which are only a few KB, the data you upload in a POST could be huge. If you're uploading a 10GB file, you don't want to park it in RAM during the hours it might take to upload to the server. This would kill your client's performance in the meantime. Instead, you need a way to get a Stream so you only have to load small chunks of data into RAM before sending to the server. And remember that Stream has no Open() method, so your API must provide an open stream.

Now you have an answer to your first question: HttpWebRequest.GetRequestStream must make the network connection because if it didn't then the stream would be closed and you couldn't write to it.

Now on to your second question: how can you delay the connection? I assume you mean that the connection should happen upon the first write to the request stream. One way to do this would be to write a class that inherits from Stream that only calls GetRequestStream as late as possible, and then delegates all methods to the underlying request stream. Something like this as as starting point:

using System.Net;
using System.Threading.Tasks;
using System.Threading;

class DelayConnectRequestStream : Stream 
{
  private HttpWebRequest _req;
  private Stream _stream = null;

  public DelayConnectRequestStream (HttpWebRequest req) 
  {
    _req = req;
  }

  public void Write (byte[] buffer, int offset, int count) 
  {
    if (_stream == null) 
    {
      _stream = req.GetRequestStream();
    }
    return _stream.Write(buffer, offset, count);
  }

  public override WriteAsync (byte[] buffer, int offset, int count, CancellationToken cancellationToken)
  {
    if (_stream == null) 
    {
      // TODO: figure out if/how to make this async 
      _stream = req.GetRequestStream();
    }
    return _stream.WriteAsync(buffer, offset, count, cancellationToken);
  }

  // repeat the pattern above for all needed methods on Stream 
  // you may need to decide by trial and error which properties and methods
  // must require an open stream. Some properties/methods you can probably just return
  // without opening the stream, e.g. CanRead which will always be false so no need to 
  // create a stream before returning from that getter. 

  // Also, the code sample above is not thread safe. For 
  // thread safety, you could use Lazy<T> or roll your own locking. 
} 

But honestly the approach above seems like overkill. If I were in your shoes, I'd look at why I am trying to defer opening of the stream and to see if there's another way to solve this problem.