Tracking progress of HttpWebRequest and HttpWebResponse

Question

I'm using Sharefile API that sends HTTP requests and gets their respective answers. They are made through URL's and use always the same function. Here it is.

Private Function InvokeShareFileOperation(ByVal requestUrl As String) As JObject

    Dim request As HttpWebRequest = WebRequest.Create(requestUrl)
    Dim response As HttpWebResponse = request.GetResponse()

    Dim reader As StreamReader = New StreamReader(response.GetResponseStream())

    Dim json As String = reader.ReadToEnd()
    response.Close()
    Return JObject.Parse(json)

End Function

As some operations are a bit long, I need to somehow track their progress while they are underway and don't know how to do it. Later I'm intending to use that progress and create a progress bar from it.

(EDIT) By the way, it's the second code line (below) that takes most time, that is the operation to track.

Dim response As HttpWebResponse = request.GetResponse()

Answer 1

EDIT: I don't think you are going to be able to measure progress in any accurate way here as the bulk of the operation seems to be reliant on the remote server processing the request. GetResponse() handles setting up the DNS, connecting, sending and waiting for the remote server and this is all out of your hands. Reading the response stream is only measurable if the content-length header is returned. Personally I would show progress as 20% initially, 60% when GetResponse returns, and then the last 40% could be shown incrementally if you have the content length before downloading, or done in one go once you have finished reading the response stream.

As its a web request you can find out the content length first and then read the response stream using a buffer instead of ReadToEnd(). This allows you to calculate the progress and fire off notifications while downloading the response.

Dim request As HttpWebRequest = WebRequest.Create(requestUrl)

Using response As HttpWebResponse = request.GetResponse()
  Dim contentLength As Long = response.ContentLength
  Dim bytesReceived As Long
  Dim bufferLength as Integer = 8192
  Dim buffer(bufferLength) As Char
  Dim sb As New StringBuilder

  Using reader As StreamReader = New StreamReader(response.GetResponseStream())
    Do
      Dim bufferedCount As Integer = reader.Read(buffer, 0, bufferLength)
      sb.Append(buffer, 0, bufferedCount)
      bytesReceived += bufferedCount
      Console.WriteLine(bytesReceived / contentLength * 100 & "%")
    Loop While bytesReceived < contentLength
  End Using

  Return JObject.Parse(sb.ToString)
End Using

Obviously you can substitute the Console.WriteLine with a progress update function or a call to a SignalR hub to update a web page, and you can experiment with the buffer size to see what works best for you.

Answer 2

First we must find out what's slowing down. Request isn't send until GetResponse() is called, so processing by server can take some time. Downloading can also take some time. If response is small (relative to connection speed), you can't do much (you can if server is yours, but we'll focus on client) because you can't get progress from server. If response is large, and you want to track downloading, you can only do it if you have Content-Length header. And to get only headers, server must support HEAD request method. So here is code :

Imports System
Imports System.Net
Imports System.IO
Imports System.Text
Imports System.Threading
Imports Microsoft.VisualBasic

Public Class Form1

    Private Function InvokeShareFileOperation(ByVal requestUrl As String) As JObject
        HTTPWebRequest_GetResponse.Main(requestUrl)
        ProgressBar1.Value = 0
        Dim result As String
        Do
            Try
                ProgressBar1.Value = HTTPWebRequest_GetResponse.progress
            Catch ex As ArgumentOutOfRangeException
                ProgressBar1.Style = ProgressBarStyle.Marquee
            End Try
            If HTTPWebRequest_GetResponse.done = True Then
                result = HTTPWebRequest_GetResponse.response
                ProgressBar1.Style = ProgressBarStyle.Continuous
                ProgressBar1.Value=100
                Debug.WriteLine(result)
                Return JObject.Parse(result)
                Exit Do
            End If
        Loop
    End Function

End Class


Public Class RequestState
    ' This class stores the State of the request. 
    Private BUFFER_SIZE As Integer = 1024
    Public requestData As StringBuilder
    Public BufferRead() As Byte
    Public request As HttpWebRequest
    Public response As HttpWebResponse
    Public streamResponse As Stream

    Public Sub New()
        BufferRead = New Byte(BUFFER_SIZE) {}
        requestData = New StringBuilder("")
        request = Nothing
        streamResponse = Nothing
    End Sub 'New 
End Class 'RequestState


Class HTTPWebRequest_GetResponse

    Private BUFFER_SIZE As Integer = 1024
    Public Shared response As String
    Public Shared done As Boolean = False
    Public Shared length As Long = 1
    Public Shared progress As Integer
    Public Shared myHttpWebRequest As HttpWebRequest
    Public Shared myRequestState As New RequestState()

    Shared Sub Main(url As String)

        Try
            Dim headRequest As HttpWebRequest = WebRequest.Create(url)
            headRequest.Method = "HEAD"
            Dim headResponse As HttpWebResponse = headRequest.GetResponse
            length = headResponse.ContentLength
            Debug.WriteLine(length)
            headResponse.Close()
            ' Create a HttpWebrequest object to the desired URL.  
            myHttpWebRequest = WebRequest.Create(url)

            ' Create an instance of the RequestState and assign the previous myHttpWebRequest 
            ' object to its request field.   

            myRequestState.request = myHttpWebRequest
            'Dim myResponse As New HTTPWebRequest_GetResponse()

            ' Start the asynchronous request. 
            Dim result As IAsyncResult = CType(myHttpWebRequest.BeginGetResponse(New AsyncCallback(AddressOf RespCallback), myRequestState), IAsyncResult)

        Catch e As WebException
            Debug.WriteLine("Main Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("Main Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'Main

    Private Shared Sub RespCallback(asynchronousResult As IAsyncResult)
        Debug.WriteLine("RespCallBack entered")
        Try
            ' State of request is asynchronous. 
            Dim myRequestState As RequestState = CType(asynchronousResult.AsyncState, RequestState)
            Dim myHttpWebRequest As HttpWebRequest = myRequestState.request
            myRequestState.response = CType(myHttpWebRequest.EndGetResponse(asynchronousResult), HttpWebResponse)

            ' Read the response into a Stream object. 
            Dim responseStream As Stream = myRequestState.response.GetResponseStream()
            myRequestState.streamResponse = responseStream

            ' Begin the Reading of the contents of the HTML page. 
            Dim asynchronousInputRead As IAsyncResult = responseStream.BeginRead(myRequestState.BufferRead, 0, 1024, New AsyncCallback(AddressOf ReadCallBack), myRequestState)
            Return
        Catch e As WebException
            Debug.WriteLine("RespCallback Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("RespCallback Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'RespCallback

    Private Shared Sub ReadCallBack(asyncResult As IAsyncResult)
        Debug.WriteLine("ReadCallBack entered")
        Try

            Dim myRequestState As RequestState = CType(asyncResult.AsyncState, RequestState)
            Dim responseStream As Stream = myRequestState.streamResponse
            Dim read As Integer = responseStream.EndRead(asyncResult)
            ' Read the HTML page. 
            If read > 0 Then
                myRequestState.requestData.Append(Encoding.ASCII.GetString(myRequestState.BufferRead, 0, read))
                If length = -1 Or length = 0 Then
                    progress = -1
                Else
                    progress = myRequestState.BufferRead.Length * 100 / length
                    Debug.WriteLine(progress)
                End If
                Dim asynchronousResult As IAsyncResult = responseStream.BeginRead(myRequestState.BufferRead, 0, 1024, New AsyncCallback(AddressOf ReadCallBack), myRequestState)

            Else
                If myRequestState.BufferRead.Length > 1 Then
                    Dim fullResponse As String = myRequestState.requestData.ToString
                    response = fullResponse.Substring(0, fullResponse.IndexOf("</body>")).Substring(fullResponse.IndexOf(">", fullResponse.IndexOf("<body")) + 2) 'Returns only body
                    ' Release the HttpWebResponse resource.
                    myRequestState.response.Close()
                    done = True
                    Debug.WriteLine(done)
                End If

                responseStream.Close()
            End If

        Catch e As WebException
            Debug.WriteLine("ReadCallBack Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("ReadCallBack Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'ReadCallBack 
End Class 'HttpWebRequest_BeginGetResponse

I took code from http://msdn.microsoft.com/en-us/library/debx8sh9(v=vs.110).aspx and changed it.

EDIT: Code now returns only body and response is closed.

EDIT2: As @Geezer68 said, it's not 100% accurate, but it's OK for showing progress to user.

Answer 3

I'm pretty sure what you want is reader.BaseStream.Length so you can know the length before reading. (At least I did, so I tried) But it threw a NotSupportedException with the message This stream does not support seek operations. So I googled StreamReader + This stream... and found this SO link:

Error “This stream does not support seek operations” in C#

So the short answer is: It is not possible.