Uploading HTTP progress tracking

Question

I've got WPF application I'm writing that posts files to one of social networks. Upload itself working just fine, but I'd like to provide some indication of how far along I am with the uploading.

I tried a bunch of ways to do this:

1) HttpWebRequest.GetStream method:

using (
 var FS = File.Open(
  localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Method = "POST";
    request.ProtocolVersion = HttpVersion.Version11;
    request.ContentType = "multipart/form-data; boundary=--AaB03x";
    //predata and postdata is two byte[] arrays, that contains
    //strings for MIME file upload (defined above and is not important)
    request.ContentLength = predata.Length + FS.Length + postdata.Length;
    request.AllowWriteStreamBuffering = false;
    using (var reqStream = request.GetRequestStream())
    {
        reqStream.Write(predata, 0, predata.Length);
        int bytesRead = 0;
        int totalRead = 0;
        do
        {
            bytesRead = FS.Read(fileData, 0, MaxContentSize);
            totalRead += bytesRead;
            reqStream.Write(fileData, 0, bytesRead);
            reqStream.Flush(); //trying with and without this
            //this part will show progress in percents
            sop.prct = (int) ((100*totalRead)/len);
        } while (bytesRead > 0);
        reqStream.Write(postdata, 0, postdata.Length);
    }
    HttpWebResponse responce = (HttpWebResponse) request.GetResponse();
    using (var respStream = responce.GetResponseStream())
    {
        //do things
    }
}

2) WebClient way (much shorter):

void UploadFile (url, localFilePath)
{
    ...
    WebClient client = new WebClient();
    client.UploadProgressChanged += new UploadProgressChangedEventHandler(UploadPartDone);
    client.UploadFileCompleted += new UploadFileCompletedEventHandler(UploadComplete);
    client.UploadFileAsync(new Uri(url), localFilePath);
    done.WaitOne();

    //do things with responce, received from UploadComplete
    JavaScriptSerializer jssSer = new JavaScriptSerializer();
    return jssSer.Deserialize<UniversalJSONAnswer>(utf8.GetString(UploadFileResponce));
    //so on...
    ...
}

void UploadComplete(object sender, UploadFileCompletedEventArgs e)
{
    UploadFileResponce=e.Result;
    done.Set();
}

void UploadPartDone(object sender, UploadProgressChangedEventArgs e)
{
    //this part expected to show progress
    sop.prct=(int)(100*e.BytesSent/e.TotalBytesToSend);
}

3) Even TcpClient way:

using (
 var FS = File.Open(
  localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    long totalRead = 0;
    using (var client = new TcpClient(urli.Host, urli.Port))
    {
        using (var clearstream = client.GetStream())
        {
            using (var writer = new StreamWriter(clearstream))
            using (var reader = new StreamReader(clearstream))
            {
                //set progress to 0
                sop.prct = 0;
                // Send request headers
                writer.WriteLine("POST " + urli.AbsoluteUri + " HTTP/1.1");
                writer.WriteLine("Content-Type: multipart/form-data; boundary=--AaB03x");
                writer.WriteLine("Host: " + urli.Host);
                writer.WriteLine("Content-Length: " + (predata.Length + len + postdata.Length).ToString());
                writer.WriteLine();
                //some data for MIME
                writer.Write(utf8.GetString(predata));
                writer.Flush();
                int bytesRead;
                do
                {
                    bytesRead = FS.Read(fileData, 0, MaxContentSize);
                    totalRead += bytesRead;
                    writer.BaseStream.Write(fileData, 0, bytesRead);
                    writer.BaseStream.Flush();
                    sop.prct = (int) ((100*totalRead)/len);
                } while (bytesRead > 0)
                writer.Write(utf8.GetString(postdata));
                writer.Flush();
                //read line of response and do other thigs...
                respStr = reader.ReadLine();
                ...
            }
        }
    }
}

In all cases the file was successfully sent to the server. But always progress looks like this: for a few seconds it runs from 0 to 100 and then waits until file actually uploading (about 5 minutes - file is 400MB).

So I think the data from a file is buffered somewhere and I'm tracking not uploading, but buffering data. And then must wait until it's uploaded.

My questions are:

1) Is there any way to track actual uploading data? That the method Stream.Write() or Flush() (which as I read somewhere, does not work for NetworkStream) did not return until it receives confirmation from the server that the TCP packets received.

2) Or can I deny buffering (AllowWriteStreamBUffering for HttpWebRequest doesn't work)?

3) And does it make sense to go further "down" and try with Sockets?

updated:

To avoid any doubts in the way of progress displaying on UI, I rewrote the code to log a file. so, here is code:

using (var LogStream=File.Open("C:\\123.txt",FileMode.Create,FileAccess.Write,FileShare.Read))
using (var LogWriter=new StreamWriter(LogStream))
using (var FS = File.Open(localFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
    long len = FS.Length;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Timeout = 7200000; //2 hour timeout
    request.Method = "POST";
    request.ProtocolVersion = HttpVersion.Version11;
    request.ContentType = "multipart/form-data; boundary=--AaB03x";
    //predata and postdata is two byte[] arrays, that contains
    //strings for MIME file upload (defined above and is not important)
    request.ContentLength = predata.Length + FS.Length + postdata.Length;
    request.AllowWriteStreamBuffering = false;
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " Start write into request stream. ");
    using (var reqStream = request.GetRequestStream())
    {
        reqStream.Write(predata, 0, predata.Length);
        int bytesRead = 0;
        int totalRead = 0;
        do
        {
            bytesRead = FS.Read(fileData, 0, MaxContentSize);
            totalRead += bytesRead;
            reqStream.Write(fileData, 0, bytesRead);
            reqStream.Flush(); //trying with and without this
            //sop.prct = (int) ((100*totalRead)/len); //this part will show progress in percents
            LogWriter.WriteLine(DateTime.Now.ToString("o") + " totalRead= " + totalRead.ToString() + " / " + len.ToString());
        } while (bytesRead > 0);
        reqStream.Write(postdata, 0, postdata.Length);
    }
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " All sent!!! Waiting for responce... ");
    LogWriter.Flush();
    HttpWebResponse responce = (HttpWebResponse) request.GetResponse();
    LogWriter.WriteLine(DateTime.Now.ToString("o") + " Responce received! ");
    using (var respStream = responce.GetResponseStream())
    {
        if (respStream == null) return null;
        using (var streamReader = new StreamReader(respStream))
        {
            string resp = streamReader.ReadToEnd();
            JavaScriptSerializer jssSer = new JavaScriptSerializer();
            return jssSer.Deserialize<UniversalJSONAnswer>(resp);
        }
    }
}

and here is result (I cut the middle):

2011-11-19T22:00:54.5964408+04:00 Start write into request stream. 
2011-11-19T22:00:54.6404433+04:00 totalRead= 1048576 / 410746880
2011-11-19T22:00:54.6424434+04:00 totalRead= 2097152 / 410746880
2011-11-19T22:00:54.6434435+04:00 totalRead= 3145728 / 410746880
2011-11-19T22:00:54.6454436+04:00 totalRead= 4194304 / 410746880
2011-11-19T22:00:54.6464437+04:00 totalRead= 5242880 / 410746880
2011-11-19T22:00:54.6494438+04:00 totalRead= 6291456 / 410746880
.......    
2011-11-19T22:00:55.3434835+04:00 totalRead= 408944640 / 410746880
2011-11-19T22:00:55.3434835+04:00 totalRead= 409993216 / 410746880
2011-11-19T22:00:55.3464837+04:00 totalRead= 410746880 / 410746880
2011-11-19T22:00:55.3464837+04:00 totalRead= 410746880 / 410746880
2011-11-19T22:00:55.3464837+04:00 All sent!!! Waiting for responce... 
2011-11-19T22:07:23.0616597+04:00 Responce received! 

as you can see program thinks that it uploaded ~400MB for about 2 seconds. And after 7 minutes file actually uploads and I receive responce.

updated again:

Seems to this is happening under WIndows 7 (not shure about x64 or x86). When I run my code uder XP everything works perfectly and progress is shown absolute correctly

Answer 1

my suggestion is to use new HTTPClient class (available in .NET 4.5). It supports progress.

This article helped me a lot with this: http://www.strathweb.com/2012/06/drag-and-drop-files-to-wpf-application-and-asynchronously-upload-to-asp-net-web-api/

My code for upload file:

    private void HttpSendProgress(object sender, HttpProgressEventArgs e)
    {
        HttpRequestMessage request = sender as HttpRequestMessage;
        Console.WriteLine(e.BytesTransferred);
    }

    private void Window_Loaded_1(object sender, RoutedEventArgs e)
    {
        ProgressMessageHandler progress = new ProgressMessageHandler();
        progress.HttpSendProgress += new EventHandler<HttpProgressEventArgs>(HttpSendProgress);

        HttpRequestMessage message = new HttpRequestMessage();
        StreamContent streamContent = new StreamContent(new FileStream("e:\\somefile.zip", FileMode.Open));

        message.Method = HttpMethod.Put;
        message.Content = streamContent;
        message.RequestUri = new Uri("{Here your link}");

        var client = HttpClientFactory.Create(progress);

        client.SendAsync(message).ContinueWith(task =>
        {
            if (task.Result.IsSuccessStatusCode)
            { 

            }
        });
    }

Answer 2

You could use the WebClient's UploadFile to upload file rather than using writing file as a file stream. In order to track the percentage of the data received and uploaded you can use UploadFileAsyn and subscribe to its events.

In the code bellow I've used UploadFileAsyn to the upload files synchronously, but it need not to be synchronous as far as you don't dispose the instance of the uploader.

class FileUploader : IDisposable
{
    private readonly WebClient _client;
    private readonly Uri _address;
    private readonly string _filePath;
    private bool _uploadCompleted;
    private bool _uploadStarted;
    private bool _status;

    public FileUploader(string address, string filePath)
    {
        _client = new WebClient();
        _address = new Uri(address);
        _filePath = filePath;
        _client.UploadProgressChanged += FileUploadProgressChanged;
        _client.UploadFileCompleted += FileUploadFileCompleted;
    }

    private void FileUploadFileCompleted(object sender, UploadFileCompletedEventArgs e)
    {
        _status = (e.Cancelled || e.Error == null) ? false : true;
        _uploadCompleted = true;
    }

    private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
    {
        if(e.ProgressPercentage % 10 == 0)
        {
            //This writes the pecentage data uploaded and downloaded
            Console.WriteLine("Send: {0}, Received: {1}", e.BytesSent, e.BytesReceived);
            //You can have a delegate or a call back to update your UI about the percentage uploaded
            //If you don't have the condition (i.e e.ProgressPercentage % 10 == 0 )for the pecentage of the process 
            //the callback will slow you upload process down
        }
    }

    public bool Upload()
    {

        if (!_uploadStarted)
        {
            _uploadStarted = true;
            _client.UploadFileAsync(_address, _filePath);
        }
        while (!_uploadCompleted)
        {
            Thread.Sleep(1000);
        }
        return _status;
    }

    public void Dispose()
    {
        _client.Dispose();
    }
}

Client Code:

            using (FileUploader uploader = new FileUploader("http://www.google.com", @"C:\test.txt"))
        {
            uploader.Upload();
        }

You can register a custom callback (may be a delegate) on the FileUploadProgressChanged event handler to update your WPF UI.

The upload progress changed event get called more often if your callback for the event does any IO then that'll slowdown the download progress. It's best to have infrequent update e.g. the following code update only evey 10% up.

    private int _percentageDownloaded;

    private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
    {
        if (e.ProgressPercentage % 10 == 0 && e.ProgressPercentage > _percentageDownloaded)
        {

            _percentageDownloaded = e.ProgressPercentage;
            //Any callback instead of printline
            Console.WriteLine("Send: {0} Received: {1}", e.BytesSent, e.BytesReceived);
        }
    }

Answer 3

This one has been bugging me for at least one day. I have started with using WebClient.UploadFileAsync, next tried the ProgressMessageHandler for HttpClient then rolled my own HttpContent for the HttpClient API. None of those approaches worked (for me).

It appears HttpWebRequest, which sits at the bottom of most (all?) .NET Http abstraction like WebClient and HttpClient, buffers the request and response stream by default, which I confirmed by looking at it in ILSpy.

As others have noted, you can make your request use chunked encoding one way or another which will effectively disable buffering the request stream, but still this is not going to fix the progress reporting.

I found that it was necessary to flush the request stream after each block that I send in order to accurately reflect sending progress, or else your data will simply be buffered one step further down the pipeline (probably somewhere in NetworkStream or OS, didn't check). The sample code below works for me and also does a minimalistic job at translating back from a HttpWebResponse to HttpResponseMessage (which you may not need, YMMV).

public async Task<HttpResponseMessage> UploadFileAsync( string uploadUrl, string absoluteFilePath, Action<int> progressPercentCallback )
    {
        var length = new FileInfo( absoluteFilePath ).Length;

        var request = new HttpWebRequest( new Uri(uploadUrl) ) {
            Method = "PUT",
            AllowWriteStreamBuffering = false,
            AllowReadStreamBuffering = false,
            ContentLength = length
        };

        const int chunkSize = 4096;
        var buffer = new byte[chunkSize];

        using (var req = await request.GetRequestStreamAsync())
        using (var readStream = File.OpenRead(absoluteFilePath))
        {
            progressPercentCallback(0);
            int read = 0;
            for (int i = 0; i < length; i += read)
            {
                read = await readStream.ReadAsync( buffer, 0, chunkSize );
                await req.WriteAsync( buffer, 0, read );
                await req.FlushAsync(); // flushing is required or else we jump to 100% very fast
                progressPercentCallback((int)(100.0 * i / length));
            }
            progressPercentCallback(100);
        }

        var response = (HttpWebResponse)await request.GetResponseAsync();
        var result = new HttpResponseMessage( response.StatusCode );
        result.Content = new StreamContent( response.GetResponseStream() );

        return result; 
    }