How to run multiple tasks, handle exceptions and still return results

Question

I am updating my concurrency skillset. My problem seems to be fairly common: read from multiple Uris, parse and work with the result, etc. I have Concurrency in C# Cookbook. There are a few examples using GetStringAsync, such as

static async Task<string> DownloadAllAsync(IEnumerable<string> urls)
{
    var httpClient = new HttpClient();

    var downloads = urls.Select(url => httpClient.GetStringAsync(url));

    Task<string>[] downloadTasks = downloads.ToArray();

    string[] htmlPages = await Task.WhenAll(downloadTasks);

    return string.Concat(htmlPages);
}

What I need is the asynchronous pattern for running multiple async tasks, capturing full or partial success.

1. Url 1 succeeds
2. Url 2 succeeds
3. Url 3 fails (timeout, bad Uri format, 401, etc)
4. Url 4 succeeds
5. ... 20 more with mixed success

waiting on DownloadAllAsync task will throw a single aggregate exception if any fail, dropping the accumulated results. From my limited research, with WhenAll or WaitAll behave the same. I want to catch the exceptions, log the failures, but continue with the remaining tasks, even if they all fail. I could process them one by one, but wouldn't that defeat the purpose of allowing TPL to manage the whole process? Is there a link to a pattern which would accomplish this in a pure TPL way? Perhaps I'm using the wrong tool?

Answer 1

I want to catch the exceptions, log the failures, but continue with the remaining tasks, even if they all fail.

In this case, the cleanest solution is to change what your code does for each element. I.e., this current code:

var downloads = urls.Select(url => httpClient.GetStringAsync(url));

says "for each url, download a string". What you want it to say is "for each url, download a string and then log and ignore any errors":

static async Task<string> DownloadAllAsync(IEnumerable<string> urls)
{
  var httpClient = new HttpClient();
  var downloads = urls.Select(url => TryDownloadAsync(httpClient, url));
  Task<string>[] downloadTasks = downloads.ToArray();
  string[] htmlPages = await Task.WhenAll(downloadTasks);
  return string.Concat(htmlPages);
}

static async Task<string> TryDownloadAsync(HttpClient client, string url)
{
  try
  {
    return await client.GetStringAsync(url);
  }
  catch (Exception ex)
  {
    Log(ex);
    return string.Empty; // or whatever you prefer
  }
}

Answer 2

You can attach continuation for all your tasks and wait for them instead of waiting directly on the tasks.

static async Task<string> DownloadAllAsync(IEnumerable<string> urls)
{
    var httpClient = new HttpClient();

    IEnumerable<Task<Task<string>>> downloads = urls.Select(url => httpClient.GetStringAsync(url).ContinueWith(p=> p, TaskContinuationOptions.ExecuteSynchronously));

    Task<Task<string>>[] downloadTasks = downloads.ToArray();

    Task<string>[] compleTasks =  await Task.WhenAll(downloadTasks);

    foreach (var task in compleTasks)
    {
        if (task.IsFaulted)//Or task.IsCanceled
        {
            //Handle it
        }
    }

    var htmlPages = compleTasks.Where(x => x.Status == TaskStatus.RanToCompletion)
        .Select(x => x.Result);

    return string.Concat(htmlPages);
}

This will not stop as soon as one task fails, rather it will wait for all the tasks to complete. Then handle the success and failure separately.