Why did my async with ContinueWith deadlock?

Question

I'm not after a solution here, more an explanation of what's going on. I've refactored this code to prevent this problem but I'm intrigued why this call deadlocked. Basically I have a list of head objects and I need to load each ones details from a DB repository object (using Dapper). I attempted to do this using ContinueWith but it failed:

List<headObj> heads = await _repo.GetHeadObjects();
var detailTasks = heads.Select(s => _changeLogRepo.GetDetails(s.Id)
    .ContinueWith(c => new ChangeLogViewModel() {
         Head = s,
         Details = c.Result
 }, TaskContinuationOptions.OnlyOnRanToCompletion));

await Task.WhenAll(detailTasks);

//deadlock here
return detailTasks.Select(s => s.Result);

Can someone explain what caused this deadlock? I tried to get my head round what has happened here but I'm not sure. I'm presuming it's something to do with calling .Result in the ContinueWith

Additional information

• This is a webapi app called in an async context

• The repo calls are all along the lines of:

  public async Task<IEnumerable<ItemChangeLog>> GetDetails(int headId)
  {
      using(SqlConnection connection = new SqlConnection(_connectionString))
      {
          return await connection.QueryAsync<ItemChangeLog>(@"SELECT [Id]
           ,[Description]
           ,[HeadId]
              FROM [dbo].[ItemChangeLog]
              WHERE HeadId = @headId", new { headId });
      }
  }
  

• I have since fixed this issue with the following code:

   List<headObj> heads = await _repo.GetHeadObjects();
   Dictionary<int, Task<IEnumerable<ItemChangeLog>>> tasks = new Dictionary<int, Task<IEnumerable<ItemChangeLog>>>();
   //get details for each head and build the vm
   foreach(ItemChangeHead head in heads)
   {
         tasks.Add(head.Id, _changeLogRepo.GetDetails(head.Id));
   }
   await Task.WhenAll(tasks.Values);
  
   return heads.Select(s => new ChangeLogViewModel() {
          Head = s,
          Details = tasks[s.Id].Result
      });
  

Answer 1

The issue is actually a combination of the above. An enumeration of tasks was created where each time the enumeration is iterated, a fresh GetDetails call. A ToList call on this Select would fix the deadlock. Without solidifying the results of the enumerable (putting them in a list), the WhenAll call evaluates the enumerable and waits for the resulting tasks asynchronously without issue, but when the returned Select statement evaluates, it's iterating and synchronously waiting on the results of tasks resulting from fresh GetDetails and ContinueWith calls that have not yet completed. All of this synchronous waiting is likely occurring while trying to serialize the response.

As to why that synchronous wait causes a deadlock, the mystery is in how await does things. It completely depends on what you're calling. An await is actually just retrieval of an awaiter via any scope-visible qualifying GetAwaiter method and registration of a callback that immediately calls GetResult on the awaiter when the work is complete. A qualifying GetAwaiter method can be an instance or extension method that returns an object having an IsCompleted property, a parameterless GetResult method (any return type, including void - result of await), and either INotifyCompletion or ICriticalNotifyCompletion interfaces. The interfaces both have OnComplete methods to register the callback. There's a mind-boggling chain of ContinueWith and await calls going on here and much of it depends on the runtime environment. The default behavior of the await you get from a Task<T> is to use SynchronizationContext.Current (I think via TaskScheduler.Current) to invoke the callback or, if that's null to use the thread pool (I think via TaskScheduler.Default) to invoke the callback. A method containing an await gets wrapped as a Task by some CompilerServices class (forgot the name), giving callers of the method the above described behaviour wrapping whatever implementation you are awaiting.

A SynchronizationContext can also customize this, but typically each context invokes on it's own single thread. If such an implementation is present on SynchronizationContext.Current when await is called on a Task, and you synchronously wait for the Result (which itself is contingent on an invoke to the waiting thread), you get a deadlock.

On the other hand, if you broke your as-is method out to another thread, or call ConfigureAwait on any of the tasks, or hide the current scheduler for your ContinueWith calls, or set your own SynchronizationContext.Current (not recommended), you change all the above.