Why is Enumerator.MoveNext not working as I expect it when used with using and async-await?

Question

I would like to enumerate through a List<int> and call a async method.

If I do this in this way:

public async Task NotWorking() {   var list = new List<int> {1, 2, 3};    using (var enumerator = list.GetEnumerator()) {     Trace.WriteLine(enumerator.MoveNext());     Trace.WriteLine(enumerator.Current);      await Task.Delay(100);   } } 

the result is:

True 0 

but I expect it to be:

True 1 

If i remove the using or the await Task.Delay(100):

public void Working1() {   var list = new List<int> {1, 2, 3};    using (var enumerator = list.GetEnumerator()) {     Trace.WriteLine(enumerator.MoveNext());     Trace.WriteLine(enumerator.Current);   } }  public async Task Working2() {   var list = new List<int> {1, 2, 3};    var enumerator = list.GetEnumerator();   Trace.WriteLine(enumerator.MoveNext());   Trace.WriteLine(enumerator.Current);    await Task.Delay(100); } 

the output is as expected:

True 1 

Can anyone explain that behavior to me?

Answer 1

Here's the short of this problem. A longer explanation follows.

• List<T>.GetEnumerator() returns a struct, a value type.
• This struct is mutable (always a recipe for disaster)
• When the using () {} is present, the struct is stored in a field on the underlying generated class to handle the await part.
• When calling .MoveNext() through this field, a copy of the field value is loaded from the underlying object, thus it is as though MoveNext was never called when the code reads .Current

As Marc mentioned in the comments, now that you know of the problem, a simple "fix" is to rewrite the code to explicitly box the struct, this will make sure the mutable struct is the same one used everywhere in this code, instead of fresh copies being mutated all over the place.

using (IEnumerator<int> enumerator = list.GetEnumerator()) { 

---

So, what happens really here.

The async / await nature of a method does a few things to a method. Specifically, the entire method is lifted onto a new generated class and turned into a state machine.

Everywhere you see await, the method is sort of "split" so that the method has to be executed sort of like this:

1. Call initial part, up until the first await
2. The next part will have to be handled by a MoveNext sort of like an IEnumerator
3. The next part, if any, and all subsequent parts, are all handled by this MoveNext part

This MoveNext method is generated on this class, and the code from the original method is placed inside it, piecemeal to fit the various sequencepoints in the method.

As such, any local variables of the method has to survive from one call to this MoveNext method to the next, and they are "lifted" onto this class as private fields.

The class in the example can then very simplistically be rewritten to something like this:

public class <NotWorking>d__1 {     private int <>1__state;     // .. more things     private List<int>.Enumerator enumerator;      public void MoveNext()     {         switch (<>1__state)         {             case 0:                 var list = new List<int> {1, 2, 3};                 enumerator = list.GetEnumerator();                 <>1__state = 1;                 break;              case 1:                 var dummy1 = enumerator;                 Trace.WriteLine(dummy1.MoveNext());                 var dummy2 = enumerator;                 Trace.WriteLine(dummy2.Current);                 <>1__state = 2;                 break; 

This code is nowhere near the correct code, but close enough for this purpose.

The problem here is that second case. For some reason the code generated reads this field as a copy, and not as a reference to the field. As such, the call to .MoveNext() is done on this copy. The original field value is left as-is, so when .Current is read, the original default value is returned, which in this case is 0.

---

So let's look at the generated IL of this method. I executed the original method (only changing Trace to Debug) in LINQPad since it has the ability to dump the IL generated.

I won't post the whole IL code here, but let's find the usage of the enumerator:

Here's var enumerator = list.GetEnumerator():

IL_005E:  ldfld       UserQuery+<NotWorking>d__1.<list>5__2 IL_0063:  callvirt    System.Collections.Generic.List<System.Int32>.GetEnumerator IL_0068:  stfld       UserQuery+<NotWorking>d__1.<enumerator>5__3 

And here's the call to MoveNext:

IL_007F:  ldarg.0      IL_0080:  ldfld       UserQuery+<NotWorking>d__1.<enumerator>5__3 IL_0085:  stloc.3     // CS$0$0001 IL_0086:  ldloca.s    03 // CS$0$0001 IL_0088:  call        System.Collections.Generic.List<System.Int32>+Enumerator.MoveNext IL_008D:  box         System.Boolean IL_0092:  call        System.Diagnostics.Debug.WriteLine 

ldfld here reads the field value and pushes the value on the stack. Then this copy is stored in a local variable of the .MoveNext() method, and this local variable is then mutated through a call to .MoveNext().

Since the end result, now in this local variable, is newer stored back into the field, the field is left as-is.

---

Here is a different example which makes the problem "clearer" in the sense that the enumerator being a struct is sort of hidden from us:

async void Main() {     await NotWorking(); }  public async Task NotWorking() {     using (var evil = new EvilStruct())     {         await Task.Delay(100);         evil.Mutate();         Debug.WriteLine(evil.Value);     } }  public struct EvilStruct : IDisposable {     public int Value;     public void Mutate()     {         Value++;     }      public void Dispose()     {     } } 

This too will output 0.

Answer 2

Looks like a bug in the old compiler, possibly caused by some interference of code transformations performed in using and async.

Compiler shipping with VS2015 seems to get this correctly.