Understand HttpWebRequest in KeepAlive mode

Question

I know that the topic has been discussed many times but I need to understand how to write code in the correct way.

I use more times the same HttpWebRequest (to the same url) with protocol version HTTP 1.1.

Method = "POST"
KeepAlive = True

But every time I need to send a different request, and get a different response.

(NB. This next code it's not correct and throw an exception)

Private Sub SendHttpWebReq()
    Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)
    httpWebReq.Method = "POST"
    httpWebReq.KeepAlive = True
    httpWebReq.ContentType = "application/x-www-form-urlencoded"
    Dim myRequestString As New List(Of String) From {"abc", "def"}
    Dim ContentList As New List(Of String)
    For a = 0 To 1
        Dim inputData As String = MyRequestString(a)
        Dim postData As String = "firstone" + ChrW(61) + inputData
        Dim encoding As New System.Text.ASCIIEncoding()
        Dim byteData As Byte() = encoding.GetBytes(postData)
        httpWebReq.ContentLength = byteData.Length
        Dim newStream As IO.Stream = httpWebReq.GetRequestStream()
        newStream.Write(byteData, 0, byteData.Length)
        newStream.Flush()
        newStream.Dispose()
        Dim Response As Net.WebResponse = httpWebReq.GetResponse()
        Dim ResponseStream As Io.Stream = Response.GetResponseStream()
        Dim Content = New Io.MemoryStream()
        ResponseStream.CopyTo(Content)
        Response.Close()
        Response.Dispose()
        ResponseStream.Flush()
        ResponseStream.Dispose()
        ContentList.Add(System.Text.Encoding.UTF8.GetString(Content.ToArray))
        Content = Nothing
    Next
End Sub

When I run the code, the first time I get the correct response, but when I try to reuse the HttpWebRequest, an Exception it's thrown at this line:

httpWebReq.ContentLength = byteData.Length

the Exception is This property cannot be set after writing has started

Searching, I've found this topic:
Am I able to reuse a HttpWebRequest?

Where it's explained that to reuse a HttpWebRequest, the Stream and WebResponse must be closed, and I did it, releasing the resources.

Also in this topic it's explained the same thing:
Reusing HttpWebRequest Object

But in this other topic:
This property cannot be set after writing has started! on a C# WebRequest Object

A member says that it's not possible to reuse the HttpWebRequest.
I'm in confusion between reuse and create a new one and I need to understand what KeepAlive it's referred to: to the Connection, or to the Request?

I suppose that when I execute this instruction:

Dim httpWebReq = CType(Net.WebRequest.Create("http://www.contoso.com/"), Net.HttpWebRequest)

I should create an instance of HttpWebRequest class, but I should establish the connection with this instruction:

Dim newStream As IO.Stream = httpWebReq.GetRequestStream()

Am I correct?

Answer 1

This is what I think needs some clarifications, because this statement may be considered misleading, the way it's phrased:

1 WebRequest => 1 WebResponse. You can't change anything in a WebRequest once it has been initialized.

This remains valid, in principle, but the initialized term could be confusing. A better definition, would be:

You can't change any parameter of a WebRequest after the request has been issued and a WebResponse has been returned, until after the WebResponse is closed (disposed).

After a WebResponse has returned a result, it can be closed - explicitly or implicitly (in a Using block) - and you can request another, modifying as necessary the WebRequest parameters (e.g. changing the Method from POST to GET).
More, a WebRequest must be re-initialized before requesting a new WebResponse. If you don't, it just falls back to it's defaults.

The code I posted below, is an example of a classic context (a Web LogIn request) when a WebRequest must be re-initialized multiple times in the same procedure, to receive an undetermined number of WebResponses until a destination address (or landing page) is reached.

This is, more or less, the schema:

                  --------------
(GET or POST)     | WebRequest |       (Method is POST)
      |---------> | GET/(POST) | <-----------| <-------------- |
      |           --------------             |                 |
      |                 |                    |                 |
--------------    ---------------    ------------------   --------------
|    New     |    | WebResponse |--> | LogIn Required |-->|   LogIn    |
|  Location  |    ---------------    ------------------   |   Address  |
| (Referer   |          |                                 --------------
|    Set)    |          |
--------------     (Set Cookies)
      |                 |
      |           ---------------
      |           |    LogIn    |
 Redirection <----|     OK      |---NO---|
                  ---------------        |
                        |                |
                       YES               |
                   (Set Cookies)         |
                        |             Request
                  ---------------     Denied
                  |  Response   |        |
                  |    URI      |        |
                  ---------------        |
                        |                |
                       EXIT <------------|
                        |

Note that, issuing a WebRequest, if accessing the requested resource URI requires authentication, the server may NOT answer with a StatusCode 302 (Found), 301 (Moved) or 303 (Redirected), it might just set the StatusCode to 200 (OK). The redirection is implicit because the "Location" Header is set or, if it's a WebForm Login, the Html page retrieved contains the redirection.

Anyway, after a redirection has been detected, the new redirected location must be followed to destination. A redirection may consist of one or more Hops, which often must be addressed manually (to verify that we're sent where we actually want to go).

---

About the keep-alive Header.
A keep-alive header is set by the Client and/or the Server, to hint the counterpart that the established connection should be maintained open, at least for some time, because the chance that other resources, linked to the current transaction, will be exchanged is high. This prevents the creation of a possibly high number of costly connections.
This setup is common in Http and Ftp requests and it's the standard in Http 1.1.

Hypertext Transfer Protocol (HTTP) Keep-Alive Header (IETF)
Compatibility with HTTP/1.0 Persistent Connections (IETF)

It needs to be remembered that the keep-alive header is referring to the Connection that has been established with a WebRequest, not to the WebRequest itself. When a connection is created specifying that it should be kept open, subsequent requests should maintain the connection: keep-alive Header to conform with the protocol.
In a .NET HttpWebRequest, setting the KeepAlive property to False, is equivalent to setting the connection: close Header.

However, the logic which governs a Connection and the Connection-Pool which a process is granted access to, is managed by the ServicePointManager, using a ServicePoint as a reference for every connection request.

Thus, a WebRequest may specify that the Connection it is asking to create should be kept open (because it needs to be reused more times), but the real logic behind a Connection, how it is established, maintained and managed, lies somewhere else.

In HTTP 2.0 (StackOverflow/StackExchange use this protocol), the keep-alive setting is completely ignored, for this exact reason. A Connection logic is managed at a higher, independent, level.

(...) when you call 1 new HttpWebRequest every 3 seconds, every 10 seconds, or every 60 seconds? What's the difference when i send those requests with True or False?

You set the KeepAlive property to hint the Connection Manager that the Connection established should be kept open, because you know it will be re-used. However, both the ServicePointManager and the remote Server will comply to the request if the protocol in use provides for this setting and within the limits imposed by the internal logic that governs the complex of the Connection-Pools.
It should be set in HTTP 1.0, it's the default setting in HTTP 1.1, it's ignored in HTTP 2.0.

Since you can't know which protocol will be used until a Connection is established, it's usually set to keep-alive, because some devices (Proxies, specifically) in the route to the requested resource, might need this setting to be explicit (read the IETF documents about Proxies and their behavior).

---

In this example, a WebRequest is repeatedly initialized in a Loop, and the underlying WebResponse is disposed each time, until a StatusCode 200 (OK) is received or the request is denied or we have been redirected too many times (a Cancellation Token may also be useful here).

In the example, the main method is meant to be called this way:

Public Async Sub SomeMethodAsync()
    LoginParameters = New LoginObject() With {
        .CookieJar = New CookieContainer,
        .LogInUrl = "[Some IP Address]",
        .Credentials = New Dictionary(Of String, String)
    }
    LoginParameters.Credentials.Add("UserName", "[Username]")
    LoginParameters.Credentials.Add("Email", "[email]")
    LoginParameters.Credentials.Add("Password", "[Password]")

    LoginParameters = Await HttpLogIn(LoginParameters)
7End Sub

The LoginParameters object must be preserved, because references a CookieContainer, which contains the Cookies received after the authentication. These Cookies are passed to the server when a new WebRequest is initialized, as a "proof" that the request's credentials are already authenticated. Note that these Cookies expire after a while (these are "refreshed" when a new WebRequest is issued, unless the Session has a time limit). If this is the case, the Login procedure is automatically repeated.

Imports System.Net
Imports System.Net.Security
Imports System.IO
Imports System.Security
Imports System.Security.Cryptography
Imports System.Security.Cryptography.X509Certificates
Imports System.Text

Public LoginParameters As LoginObject

Public Class LoginObject
    Public Property LogInUrl As String
    Public Property ResponseUrl As String
    Public Property Credentials As Dictionary(Of String, String)
    Public Property StatusCode As HttpStatusCode
    Public Property CookieJar As New CookieContainer()
End Class

Public Async Function HttpLogIn(LogInParameters As LoginObject) As Task(Of LoginObject)
    Dim httpRequest As HttpWebRequest
    Dim StatusCode As HttpStatusCode
    Dim MaxHops As Integer = 20

    ' Windows 7 (.Net 4.5.1+ required):     
    'ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

    ' Windows 10 (.Net 4.5.1+ required):    
    ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault

    'If needed or for testing
    'ServicePointManager.ServerCertificateValidationCallback = AddressOf CertificateValidation

    httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)

    Try
        HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)
        Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
            StatusCode = httpResponse.StatusCode
        End Using

        If StatusCode = HttpStatusCode.OK OrElse StatusCode = HttpStatusCode.NoContent Then
            'POST Parameters are URLEncoded and the encoded strings converted to a Byte array of UTF8 chars
            Dim EncodedParameters As Byte() = HTTP_EncodePOSTParameters(LogInParameters.Credentials)

            httpRequest = WebRequest.CreateHttp(LogInParameters.LogInUrl)
            httpRequest.Method = WebRequestMethods.Http.Post
            httpRequest.ContentType = "application/x-www-form-urlencoded"
            httpRequest.ContentLength = EncodedParameters.Length
            HTTP_RequestHeadersInit(httpRequest, String.Empty, LogInParameters.CookieJar)

            Using stream As Stream = Await httpRequest.GetRequestStreamAsync()
                stream.Write(EncodedParameters, 0, EncodedParameters.Length)
            End Using

            Dim Hops As Integer = 0
            Dim Referer As String = LogInParameters.LogInUrl
            Dim LastHttpMethod As String = httpRequest.Method

            Do
                'Evaluate Authentication redirect or page moved
                Using httpResponse As HttpWebResponse = CType(Await httpRequest.GetResponseAsync(), HttpWebResponse)
                    StatusCode = httpResponse.StatusCode
                    LogInParameters.ResponseUrl = URIFromResponseLocation(httpResponse).ToString()
                End Using

                If (StatusCode = HttpStatusCode.Moved) OrElse
                   (StatusCode = HttpStatusCode.Found) OrElse
                   (StatusCode = HttpStatusCode.RedirectMethod) OrElse
                   (StatusCode = HttpStatusCode.RedirectKeepVerb) Then

                    httpRequest = WebRequest.CreateHttp(LogInParameters.ResponseUrl)
                    HTTP_RequestHeadersInit(httpRequest, Referer, LogInParameters.CookieJar)
                    If StatusCode = HttpStatusCode.RedirectKeepVerb Then
                        httpRequest.Method = LastHttpMethod
                    Else
                        LastHttpMethod = httpRequest.Method
                    End If
                End If

                If (CType(StatusCode, Integer) > 320) OrElse Hops >= MaxHops Then
                    Exit Do
                End If
                Hops += 1
            Loop While (StatusCode <> HttpStatusCode.OK)

            If StatusCode = HttpStatusCode.OK Then
                LogInParameters.CookieJar = httpRequest.CookieContainer
            End If
        End If

    Catch exW As WebException
        StatusCode = If(exW.Response IsNot Nothing,
                        CType(exW.Response, HttpWebResponse).StatusCode,
                        CType(exW.Status, HttpStatusCode))

    Catch exS As System.Exception
        StatusCode = CType(WebExceptionStatus.RequestCanceled, HttpStatusCode)

    Finally
        ServicePointManager.ServerCertificateValidationCallback = Nothing
    End Try

    LogInParameters.StatusCode = StatusCode
    Return LogInParameters
End Function

Private Sub HTTP_RequestHeadersInit(ByRef httpReq As HttpWebRequest, Referer As String, CookiesJar As CookieContainer)
    httpReq.Date = DateTime.Now
    httpReq.CookieContainer = CookiesJar
    httpReq.KeepAlive = True
    httpReq.ConnectionGroupName = Guid.NewGuid().ToString()
    httpReq.AllowAutoRedirect = False
    httpReq.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
    httpReq.ServicePoint.Expect100Continue = False
    httpReq.Referer = Referer
    httpReq.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"
    httpReq.Accept = "ext/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
    httpReq.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US;q=0.9,en;q=0.5")
    httpReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8")
    httpReq.Headers.Add(HttpRequestHeader.CacheControl, "no-cache")
End Sub

Private Function HTTP_EncodePOSTParameters(PostParameters As Dictionary(Of String, String)) As Byte()
    Dim Encoder As New System.Text.UTF8Encoding()
    Dim CredentialValues As New StringBuilder()
    Dim _first As Boolean = True

    For Each CurrentKeyPair As KeyValuePair(Of String, String) In PostParameters
        If _first = False Then CredentialValues.Append("&")
        CredentialValues.AppendFormat("{0}={1}", WebUtility.UrlEncode(CurrentKeyPair.Key),
                                                 WebUtility.UrlEncode(CurrentKeyPair.Value))
        _first = False
    Next

    Return Encoder.GetBytes(CredentialValues.ToString())
End Function

Private Function URIFromResponseLocation(Response As HttpWebResponse) As System.Uri
    Dim uri As Uri
    Dim Location As String = Response.Headers("Location")

    Try
        If uri.IsWellFormedUriString(Location, UriKind.Absolute) Then
            uri = New Uri(Location, UriKind.Absolute)
        Else
            Dim HostUri As String = Response.ResponseUri.GetComponents(UriComponents.SchemeAndServer,
                                                                        UriFormat.Unescaped) + Location
            uri = If(uri.IsWellFormedUriString(HostUri, UriKind.Absolute),
                        New Uri(HostUri),
                        New Uri(Response.ResponseUri.GetComponents(UriComponents.Scheme, UriFormat.Unescaped) +
                                Response.ResponseUri.Host + Location))
        End If
    Catch ExceptionOnInvalidUri As Exception
        uri = New Uri(Location, UriKind.Relative)
    End Try

    Return uri
End Function

Private Function CertificateValidation(sender As Object,
                                       CACert As X509Certificate,
                                       CAChain As X509Chain,
                                       PolicyErrors As SslPolicyErrors) As Boolean

    'This method, as it is, accepts a Server certificate in any case
    'It could be eventually adapted to refuse a connection (returning false) 
    'if the certificate is invalid, expired or from a untrusted path
    If (PolicyErrors = SslPolicyErrors.None) Then Return True

    'If a Certificated must be added to the Chain, uncomment the code below,
    'selecting a Certificate in the Local (or other) Storage
    'Dim MyCert As X509Certificate2 = New X509Certificate2("[localstorage]/[ca.cert]")
    'CAChain.ChainPolicy.ExtraStore.Add(MyCert)

    'CAChain.Build(MyCert)
    'For Each CACStatus As X509ChainStatus In CAChain.ChainStatus
    '    If (CACStatus.Status <> X509ChainStatusFlags.NoError) And
    '       (CACStatus.Status <> X509ChainStatusFlags.UntrustedRoot) Then
    '        Return False
    '    End If
    'Next

    Return True
End Function