Using HttpClient for Asynchronous File Downloads

Question

I have a service which returns a csv file to a POST request. I would like to download said file using asynchronous techniques. While I can get the file, my code has a couple of outstanding problems and questions:

1) Is this really asynchronous?

2) Is there a way to know the length of the content even though it is being sent in chunked format? Think progress bars).

3) How can I best monitor progress in order to hold off the program exit until all work is complete.

using System;
using System.IO;
using System.Net.Http;

namespace TestHttpClient2
{
    class Program
    {
        /*
         * Use Yahoo portal to access quotes for stocks - perform asynchronous operations.
         */

        static string baseUrl = "http://real-chart.finance.yahoo.com/";
        static string requestUrlFormat = "/table.csv?s={0}&d=0&e=9&f=2015&g=d&a=4&b=5&c=2000&ignore=.csv";

        static void Main(string[] args)
        {
            while (true) 
            {
                Console.Write("Enter a symbol to research or [ENTER] to exit: ");
                string symbol = Console.ReadLine();
                if (string.IsNullOrEmpty(symbol))
                    break;
                DownloadDataForStockAsync(symbol);
            }
        }

        static async void DownloadDataForStockAsync(string symbol)
        {
            try
            {
                using (var client = new HttpClient())
                {
                    client.BaseAddress = new Uri(baseUrl);
                    client.Timeout = TimeSpan.FromMinutes(5);
                    string requestUrl = string.Format(requestUrlFormat, symbol);

                    //var content = new KeyValuePair<string, string>[] {
                    //    };
                    //var formUrlEncodedContent = new FormUrlEncodedContent(content);

                    var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
                    var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
                    var response = sendTask.Result.EnsureSuccessStatusCode();
                    var httpStream = await response.Content.ReadAsStreamAsync();

                    string OutputDirectory = "StockQuotes";

                    if (!Directory.Exists(OutputDirectory))
                    {
                        Directory.CreateDirectory(OutputDirectory);
                    }

                    DateTime currentDateTime = DateTime.Now;
                    var filePath = Path.Combine(OutputDirectory, string.Format("{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}.csv",
                        symbol,
                        currentDateTime.Year, currentDateTime.Month, currentDateTime.Day,
                        currentDateTime.Hour, currentDateTime.Minute, currentDateTime.Second, currentDateTime.Millisecond
                        ));

                    using (var fileStream = File.Create(filePath))
                    using (var reader = new StreamReader(httpStream))
                    {
                        httpStream.CopyTo(fileStream);
                        fileStream.Flush();
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error, try again!");
            }
        }

    }
}

Answer 1

1. "Is this really asynchronous?"

Yes, mostly. The DownloadDataForStockAsync() method will return before the operation is complete, at the await response.Content.ReadAsStreamAsync() statement.

The main exception is near the end of the method, where you call Stream.CopyTo(). This isn't asynchronous, and because it's a potentially lengthy operation could result in noticeable delays. However, in a console program you won't notice, because the continuation of the method is executed in the thread pool rather than the original calling thread.

If you intend to move this code to a GUI framework, such as Winforms or WPF, you should change the statement to read await httpStream.CopyToAsync(fileStream);

2. Is there a way to know the length of the content even though it is being sent in chunked format? Think progress bars).

Assuming the server includes the Content-Length in the headers (and it should), yes. This should be possible.

Note that if you were using HttpWebRequest, the response object would have a ContentLength property giving you this value directly. You are using HttpRequestMessage here instead, which I'm less familiar with. But as near as I can tell, you should be able to access the Content-Length value like this:

long? contentLength = response.Content.Headers.ContentLength;

if (contentLength != null)
{
    // use value to initialize "determinate" progress indication
}
else
{
    // no content-length provided; will need to display progress as "indeterminate"
}

3. How can I best monitor progress in order to hold off the program exit until all work is complete.

There are lots of ways. I will point out that any reasonable way will require that you change the DownloadDataForStockAsync() method so that it returns Task and not void. Otherwise, you don't have access to the task that's created. You should do this anyway though, so that's not a big deal. :)

The simplest would be to just keep a list of all the tasks you start, and then wait on them before exiting:

static void Main(string[] args)
{
    List<Task> tasks = new List<Task>();

    while (true) 
    {
        Console.Write("Enter a symbol to research or [ENTER] to exit: ");
        string symbol = Console.ReadLine();
        if (string.IsNullOrEmpty(symbol))
            break;
        tasks.Add(DownloadDataForStockAsync(symbol));
    }

    Task.WaitAll(tasks);
}

Of course, this requires that you explicitly maintain a list of each Task object, including those which have already completed. If you intend for this to run for a long time and process a very large number of symbols, that might be prohibitive. In that case, you might prefer to use the CountDownEvent object:

static void Main(string[] args)
{
    CountDownEvent countDown = new CountDownEvent();

    while (true) 
    {
        Console.Write("Enter a symbol to research or [ENTER] to exit: ");
        string symbol = Console.ReadLine();
        if (string.IsNullOrEmpty(symbol))
            break;

        countDown.AddCount();
        DownloadDataForStockAsync(symbol).ContinueWith(task => countdown.Signal()) ;
    }

    countDown.Wait();
}

This simply increments the CountDownEvent counter for each task you create, and attaches a continuation to each task to decrement the counter. When the counter reaches zero, the event is set, allowing a call to Wait() to return.