Why does LogicalCallContext not work with async?

Question

In this question the accepted answer by Stephen Cleary says that LogicalCallContext can't work correctly with async. He also posted about it in this MSDN thread.

LogicalCallContext keeps a Hashtable storing the data sent to CallContext.LogicalGet/SetData. And it only does a shallow copy of this Hashtable. So if you store a mutable object in it, different tasks/threads will see each other's changes. This is why Stephen Cleary's example NDC program (posted on that MSDN thread) doesn't work correctly.

But AFAICS, if you only store immutable data in the Hashtable (perhaps by using immutable collections), that should work, and let us implement an NDC.

However, Stephen Cleary also said in that accepted answer:

CallContext can't be used for this. Microsoft has specifically recommended against using CallContext for anything except remoting. More to the point, the logical CallContext doesn't understand how async methods return early and resume later.

Unfortunately, that link to the Microsoft recommendation is down (page not found). So my question is, why is this not recommended? Why can't I use LogicalCallContext in this way? What does it mean to say it doesn't understand async methods? From the caller's POV they are just methods returning Tasks, no?

ETA: see also this other question. There, an answer by Stephen Cleary says:

you could use CallContext.LogicalSetData and CallContext.LogicalGetData, but I recommend you don't because they don't support any kind of "cloning" when you use simple parallelism

That seems to support my case. So I should be able to build an NDC, which is in fact what I need, just not for log4net.

I wrote some sample code and it seems to work, but mere testing doesn't always catch concurrency bugs. So since there are hints in those other posts that this may not work, I'm still asking: is this approach valid?

ETA: When I run Stephen's proposed repro from the answer below), I don't get the wrong answers he says I would, I get correct answers. Even where he said "LogicalCallContext value here is always "1"", I always get the correct value of 0. Is this perhaps due to a race condition? Anyway, I've still not reproduced any actual problem on my own computer. Here's the exact code I'm running; it prints only "true" here, where Stephen says it should print "false" at least some of the time.

private static string key2 = "key2"; private static int Storage2 {      get { return (int) CallContext.LogicalGetData(key2); }      set { CallContext.LogicalSetData(key2, value);}  }  private static async Task ParentAsync() {   //Storage = new Stored(0); // Set LogicalCallContext value to "0".   Storage2 = 0;    Task childTaskA = ChildAAsync();   // LogicalCallContext value here is always "1".   // -- No, I get 0   Console.WriteLine(Storage2 == 0);    Task childTaskB = ChildBAsync();   // LogicalCallContext value here is always "2".   // -- No, I get 0   Console.WriteLine(Storage2 == 0);    await Task.WhenAll(childTaskA, childTaskB);   // LogicalCallContext value here may be "0" or "1".   // -- I always get 0   Console.WriteLine(Storage2 == 0); }  private static async Task ChildAAsync() {   var value = Storage2; // Save LogicalCallContext value (always "0").   Storage2 = 1; // Set LogicalCallContext value to "1".    await Task.Delay(1000);   // LogicalCallContext value here may be "1" or "2".   Console.WriteLine(Storage2 == 1);    Storage2 = value; // Restore original LogicalCallContext value (always "0"). }  private static async Task ChildBAsync() {   var value = Storage2; // Save LogicalCallContext value (always "1").   Storage2 = 2; // Set LogicalCallContext value to "2".    await Task.Delay(1000);   // LogicalCallContext value here may be "0" or "2".   Console.WriteLine(Storage2 == 2);    Storage2 = value; // Restore original LogicalCallContext value (always "1"). }  public static void Main(string[] args) {   try {     ParentAsync().Wait();   }   catch (Exception e) {     Console.WriteLine(e);   } 

So my restated question is, what (if anything) is wrong with the above code?

Furthermore, when I look at the code for CallContext.LogicalSetData, it calls Thread.CurrentThread.GetMutableExecutionContext() and modifies that. And GetMutableExecutionContext says:

if (!this.ExecutionContextBelongsToCurrentScope)     this.m_ExecutionContext = this.m_ExecutionContext.CreateMutableCopy();   this.ExecutionContextBelongsToCurrentScope = true; 

And CreateMutableCopy eventually does a shallow copy of the LogicalCallContext's Hashtable that holds the user-supplied data.

So trying to understand why this code doesn't work for Stephen, is it because ExecutionContextBelongsToCurrentScope has the wrong value sometimes? If that's the case, maybe we can notice when it does - by seeing that either the current task ID or the current thread ID have changed - and manually store separate values in our immutable structure, keyed by thread + task ID. (There are performance issues with this approach, e.g. the retention of data for dead tasks, but apart from that would it work?)

Answer 1

Update: This answer is not correct for .NET 4.5. See my blog post on AsyncLocal for details.

Here's the situation (repeating several points in your question):

• LogicalCallContext will flow with async calls; you can use it to set some implicit data and read it from an async method further down your call stack.
• All copies of LogicalCallContext are shallow copies, without any way for end-user code to hook into a deep-copy kind of operation.
• When you do "simple parallelism" with async, there is only one copy of the LogicalCallContext shared between the various async methods.

LogicalCallContext does work fine if your async code is all linear:

async Task ParentAsync() {   ... = 0; // Set LogicalCallContext value to "0".    await ChildAAsync();   // LogicalCallContext value here is always "0".    await ChildBAsync();   // LogicalCallContext value here is always "0". }  async Task ChildAAsync() {   int value = ...; // Save LogicalCallContext value (always "0").   ... = 1; // Set LogicalCallContext value to "1".    await Task.Delay(1000);   // LogicalCallContext value here is always "1".    ... = value; // Restore original LogicalCallContext value (always "0"). }  async Task ChildBAsync() {   int value = ...; // Save LogicalCallContext value (always "0").   ... = 2; // Set LogicalCallContext value to "2".    await Task.Delay(1000);   // LogicalCallContext value here is always "2".    ... = value; // Restore original LogicalCallContext value (always "0"). } 

But things aren't so nice once you use what I call "simple parallelism" (starting several async methods and then using Task.WaitAll or similar). This is an example similar to my MSDN forum post (for simplicity, assume a non-parallel SynchronizationContext such as GUI or ASP.NET):

Edit: the code comments are incorrect; see the comments on this question and answer

async Task ParentAsync() {   ... = 0; // Set LogicalCallContext value to "0".    Task childTaskA = ChildAAsync();   // LogicalCallContext value here is always "1".    Task childTaskB = ChildBAsync();   // LogicalCallContext value here is always "2".    await Task.WhenAll(childTaskA, childTaskB);   // LogicalCallContext value here may be "0" or "1". }  async Task ChildAAsync() {   int value = ...; // Save LogicalCallContext value (always "0").   ... = 1; // Set LogicalCallContext value to "1".    await Task.Delay(1000);   // LogicalCallContext value here may be "1" or "2".    ... = value; // Restore original LogicalCallContext value (always "0"). }  async Task ChildBAsync() {   int value = ...; // Save LogicalCallContext value (always "1").   ... = 2; // Set LogicalCallContext value to "2".    await Task.Delay(1000);   // LogicalCallContext value here may be "0" or "2".    ... = value; // Restore original LogicalCallContext value (always "1"). } 

The problem is that the LogicalCallContext is shared between ParentAsync, ChildAAsync, and ChildBAsync, without any way to hook into or force a deep-copy operation. In the "linear" example, the context is also shared, but only one method was active at a time.

Even if the data you store in LogicalCallContext is immutable (as in my integer example), you'd still have to update the LogicalCallContext value in order to implement an NDC, and this means the sharing-without-copies problem is going to mess it up.

I have looked into this in detail, and have concluded that a solution is not possible. If you can figure one out, I'd be very happy to be proven wrong. :)

P.S. Stephen Toub pointed out that the recommendation to use CallContext only for remoting (which was given without reason, IIRC) no longer applies. We may feel free to use LogicalCallContext... if we can get it to work. ;)

Answer 2

Stephen confirms that this works on .Net 4.5 and Win8/2012. Not tested on other platforms, and known not to work on at least some of them. So the answer is that Microsoft got their game together and fixed the underlying issue in at least the most recent version of .Net and the async compiler.

So the answer is, it does work, just not on older .Net versions. (So the log4net project can't use it to provide a generic NDC.)