Why does my async method builder have to be a class or run in Debug mode?

Question

I'm trying to implement my own async method builder for a custom awaitable type. My awaitable type is just a struct containing a ValueTask<T>.

The problem is my asynchronous method builder only works when it's a class or compiled in Debug mode, not a struct and in Release mode.

Here's a minimal, reproducible example. You have to copy this code into a new console project on your local PC and run it in Release mode; .NET Fiddle apparently runs snippets in Debug mode. And of course this requires .Net 5+: https://dotnetfiddle.net/S6F9Hd

This code completes successfully when CustomAwaitableAsyncMethodBuilder<T> is a class or it is compiled in Debug mode. But it hangs and fails to complete otherwise:

class Program
{
    static async Task Main()
    {
        var expected = Guid.NewGuid().ToString();
        async CustomAwaitable<string> GetValueAsync()
        {
            await Task.Yield();
            return expected;
        }

        var actual = await GetValueAsync();

        if (!ReferenceEquals(expected, actual))
            throw new Exception();

        Console.WriteLine("Done!");
    }
}

Here is my custom awaitable type:

[AsyncMethodBuilder(typeof(CustomAwaitableAsyncMethodBuilder<>))]
public readonly struct CustomAwaitable<T>
{
    readonly ValueTask<T> _valueTask;

    public CustomAwaitable(ValueTask<T> valueTask)
    {
        _valueTask = valueTask;
    }

    public ValueTaskAwaiter<T> GetAwaiter() => _valueTask.GetAwaiter();
}

And here is my custom async method builder. Again, all I have to do to make the code run is change this from a struct to a class:

public struct CustomAwaitableAsyncMethodBuilder<T>
{
    Exception? _exception;
    bool _hasResult;
    SpinLock _lock;
    T? _result;
    TaskCompletionSource<T>? _source;

    public CustomAwaitable<T> Task
    {
        get
        {
            var lockTaken = false;
            try
            {
                _lock.Enter(ref lockTaken);
                if (_exception is not null)
                    return new CustomAwaitable<T>(ValueTask.FromException<T>(_exception));
                if (_hasResult)
                    return new CustomAwaitable<T>(ValueTask.FromResult(_result!));
                return new CustomAwaitable<T>(
                    new ValueTask<T>(
                        (_source ??= new TaskCompletionSource<T>(TaskCreationOptions.RunContinuationsAsynchronously))
                        .Task
                    )
                );
            }
            finally
            {
                if (lockTaken)
                    _lock.Exit();
            }
        }
    }

    public void AwaitOnCompleted<TAwaiter, TStateMachine>(
        ref TAwaiter awaiter,
        ref TStateMachine stateMachine)
        where TAwaiter : INotifyCompletion
        where TStateMachine : IAsyncStateMachine =>
        awaiter.OnCompleted(stateMachine.MoveNext);

    public void AwaitUnsafeOnCompleted<TAwaiter, TStateMachine>(
        ref TAwaiter awaiter,
        ref TStateMachine stateMachine)
        where TAwaiter : ICriticalNotifyCompletion
        where TStateMachine : IAsyncStateMachine =>
        awaiter.UnsafeOnCompleted(stateMachine.MoveNext);

    public static CustomAwaitableAsyncMethodBuilder<T> Create() => new()
    {
        _lock = new SpinLock(Debugger.IsAttached)
    };

    public void SetException(Exception exception)
    {
        var lockTaken = false;
        try
        {
            _lock.Enter(ref lockTaken);
            if (Volatile.Read(ref _source) is {} source)
            {
                source.TrySetException(exception);
            }
            else
            {
                _exception = exception;
            }
        }
        finally
        {
            if (lockTaken)
                _lock.Exit();
        }
    }

    public void SetResult(T result)
    {
        var lockTaken = false;
        try
        {
            _lock.Enter(ref lockTaken);
            if (Volatile.Read(ref _source) is {} source)
            {
                source.TrySetResult(result);
            }
            else
            {
                _result = result;
                _hasResult = true;
            }
        }
        finally
        {
            if (lockTaken)
                _lock.Exit();
        }
    }

    public void SetStateMachine(IAsyncStateMachine stateMachine) {}

    public void Start<TStateMachine>(ref TStateMachine stateMachine)
        where TStateMachine : IAsyncStateMachine => stateMachine.MoveNext();
}

I've noticed (by debugging this code in Release mode and setting lots of breakpoints) that when it's a struct in Release mode then this sequence of events happens:

1. CustomAwaitableAsyncMethodBuilder<T>.Create() is called exactly once
2. CustomAwaitableAsyncMethodBuilder<T>.Task property is accessed
3. CustomAwaitableAsyncMethodBuilder<T>.SetResult(T) method is invoked
4. The Task property and the SetResult method both execute concurrently
5. The SpinLock does not mutually exclude the respective blocks of code (they end up racing)
6. The effects of the code in the Task property are not visible to the code in the SetResult method, nor vice versa

#5 and #6 tell me that the Task property is accessed on one instance of the struct while the SetResult method is called on a different instance.

Why is that? What am I doing wrong?

---

I don't see much information out there for how exactly to implement an async method builder. The only things I have found are:

• Microsoft's documentation for builder types
• Microsoft's source code for AsyncTaskMethodBuilder<TResult>
• This gist

I've followed Microsoft's documentation. It even states

The builder type is a class or struct

...and also Microsoft's AsyncTaskMethodBuilder<TResult> is a struct.

There is one place where their documentation is incorrect (it states that the builder's "AwaitUnsafeOnCompleted() [method] should call awaiter.OnCompleted(action)", but AsyncTaskMethodBuilder doesn't do that). But other than that I assume their documentation is correct.

The only difference I can spot between my implementation and the two other implementations linked above is that mine stores result/exception inside the builder type itself, where they always delegate to a reference type to do that job.

---

Edit:

ILSpy says this is how my Main method's state machine is implemented:

// BrokenAsyncMethodBuilder.Program.<Main>d__0
using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading.Tasks;

[StructLayout(LayoutKind.Auto)]
[CompilerGenerated]
private struct <Main>d__0 : IAsyncStateMachine
{
    public int <>1__state;

    public AsyncTaskMethodBuilder <>t__builder;

    private <>c__DisplayClass0_0 <>8__1;

    private ValueTaskAwaiter<string> <>u__1;

    private void MoveNext()
    {
        int num = <>1__state;
        try
        {
            ValueTaskAwaiter<string> awaiter;
            if (num != 0)
            {
                <>8__1 = new <>c__DisplayClass0_0();
                <>8__1.expected = Guid.NewGuid().ToString();
                awaiter = GetValueAsync().GetAwaiter();
                if (!awaiter.IsCompleted)
                {
                    num = (<>1__state = 0);
                    <>u__1 = awaiter;
                    <>t__builder.AwaitUnsafeOnCompleted(ref awaiter, ref this);
                    return;
                }
            }
            else
            {
                awaiter = <>u__1;
                <>u__1 = default(ValueTaskAwaiter<string>);
                num = (<>1__state = -1);
            }
            string actual = awaiter.GetResult();
            if ((object)<>8__1.expected != actual)
            {
                throw new Exception();
            }
            Console.WriteLine("Done!");
        }
        catch (Exception exception)
        {
            <>1__state = -2;
            <>8__1 = null;
            <>t__builder.SetException(exception);
            return;
        }
        <>1__state = -2;
        <>8__1 = null;
        <>t__builder.SetResult();
        async CustomAwaitable<string> GetValueAsync()
        {
            await Task.Yield();
            return ((<>c__DisplayClass0_0)(object)this).expected;
        }
    }

    void IAsyncStateMachine.MoveNext()
    {
        //ILSpy generated this explicit interface implementation from .override directive in MoveNext
        this.MoveNext();
    }

    [DebuggerHidden]
    private void SetStateMachine(IAsyncStateMachine stateMachine)
    {
        <>t__builder.SetStateMachine(stateMachine);
    }

    void IAsyncStateMachine.SetStateMachine(IAsyncStateMachine stateMachine)
    {
        //ILSpy generated this explicit interface implementation from .override directive in SetStateMachine
        this.SetStateMachine(stateMachine);
    }
}

And this is how my GetValueAsync method's state machine is implemented:

// BrokenAsyncMethodBuilder.Program.<>c__DisplayClass0_0.<<Main>g__GetValueAsync|0>d
using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading.Tasks;

[StructLayout(LayoutKind.Auto)]
private struct <<Main>g__GetValueAsync|0>d : IAsyncStateMachine
{
    public int <>1__state;

    public CustomAwaitableAsyncMethodBuilder<string> <>t__builder;

    public <>c__DisplayClass0_0 <>4__this;

    private YieldAwaitable.YieldAwaiter <>u__1;

    private void MoveNext()
    {
        int num = <>1__state;
        <>c__DisplayClass0_0 <>c__DisplayClass0_ = <>4__this;
        string expected;
        try
        {
            YieldAwaitable.YieldAwaiter awaiter;
            if (num != 0)
            {
                awaiter = Task.Yield().GetAwaiter();
                if (!awaiter.IsCompleted)
                {
                    num = (<>1__state = 0);
                    <>u__1 = awaiter;
                    <>t__builder.AwaitUnsafeOnCompleted(ref awaiter, ref this);
                    return;
                }
            }
            else
            {
                awaiter = <>u__1;
                <>u__1 = default(YieldAwaitable.YieldAwaiter);
                num = (<>1__state = -1);
            }
            awaiter.GetResult();
            expected = <>c__DisplayClass0_.expected;
        }
        catch (Exception exception)
        {
            <>1__state = -2;
            <>t__builder.SetException(exception);
            return;
        }
        <>1__state = -2;
        <>t__builder.SetResult(expected);
    }

    void IAsyncStateMachine.MoveNext()
    {
        //ILSpy generated this explicit interface implementation from .override directive in MoveNext
        this.MoveNext();
    }

    [DebuggerHidden]
    private void SetStateMachine(IAsyncStateMachine stateMachine)
    {
        <>t__builder.SetStateMachine(stateMachine);
    }

    void IAsyncStateMachine.SetStateMachine(IAsyncStateMachine stateMachine)
    {
        //ILSpy generated this explicit interface implementation from .override directive in SetStateMachine
        this.SetStateMachine(stateMachine);
    }
}

I'm unable to find this call into CustomAwaitableAsyncMethodBuilder<T>.Create(), other than it rather unhelpfully says it is used by the Main method (which decompiles to be very similar to the code at the top of this question). Perhaps I'm not familiar enough with ILSpy, or perhaps it has a bug:

Answer 1

Found it! If you use ILSpy to disassemble the .dll compiled from the question's code (use the .NET Fiddle link and follow the question's instructions), and then turn ILSpy's language version down to C# 4 (which was the version before async/await was introduced), then you'll see that this is how the GetValueAsync method is implemented:

// BrokenAsyncMethodBuilder.Program.<>c__DisplayClass0_0
using System.Runtime.CompilerServices;

[AsyncStateMachine(typeof(<<Main>g__GetValueAsync|0>d))]
[return: System.Runtime.CompilerServices.Nullable(new byte[] { 0, 1 })]
internal CustomAwaitable<string> <Main>g__GetValueAsync|0()
{
    <<Main>g__GetValueAsync|0>d stateMachine = default(<<Main>g__GetValueAsync|0>d);
    stateMachine.<>t__builder = CustomAwaitableAsyncMethodBuilder<string>.Create();
    stateMachine.<>4__this = this;
    stateMachine.<>1__state = -1;
    stateMachine.<>t__builder.Start(ref stateMachine);
    return stateMachine.<>t__builder.Task;
}

You can reference the disassembly at the end of the question to learn how the <<Main>g__GetValueAsync|0>d type (the state machine for the GetValueAsync method) is implemented. You'll notice that the stateMachine.<>t__builder field is the CustomAwaitableAsyncMethodBuilder type (the async method builder in the question).

Now pay attention to what happens:

1. The async method builder is created
2. Its Start method is invoked with a reference to the state machine
   1. That calls .MoveNext() on the state machine
   2. MoveNext calls AwaitUnsafeOnCompleted() on the async method builder
   3. The async method builder calls UnsafeOnCompleted() on the awaiter (the awaiter is a YieldAwaitable.YieldAwaiter), giving it the state machine's MoveNext method as a delegate
   4. YieldAwaitable.YieldAwaiter.UnsafeOnCompleted posts the MoveNext delegate to the current synchronization context to be executed in a little bit
3. The async method builder's Task property is accessed and returned

Notice also how the next time that the state machine's MoveNext method is invoked, it will call SetResult on the async method builder.

So at this point there is a delegate for the state machine's MoveNext method floating around in the synchronization context. Which means a copy of the state machine has been boxed up and placed on the heap. And the state machine holds a CustomAwaitableAsyncMethodBuilder, which is itself a struct. So it gets copied too. If CustomAwaitableAsyncMethodBuilder were a class then it would live on the heap and its reference would be copied instead of its value.

So by the time the GetValueAsync method returns, there will be a boxed instance of CustomAwaitableAsyncMethodBuilder on the heap, and another instance of CustomAwaitableAsyncMethodBuilder on the stack.

I do not know why this does not happen in debug builds. Nor do I yet understand what to do about this.

But this explains why "the Task property is accessed on one instance of the struct while the SetResult method is called on a different instance."

---

Thinking about this a little more, I think this is just another disguise for the fact that C#'s async/await system requires heap allocation and dynamic dispatch by design.

I wrote about this in a little depth not long ago as I compared C#'s async/await with Rust's implementation. In that article I pointed out how C#'s awaitable expressions are composed with dynamic dispatch on heap objects (the compiler packages up a continuation/state machine step as an Action, and Action is a reference type). But in Rust, awaitable expressions are composed with static dispatch (which doesn't necessarily require the heap).

This is exactly the same thing, just looking at it from the async method builder's perspective instead of the awaitable expression's perspective. Continuations are packaged up as state machine steps (the MoveNext method). It doesn't matter if the state machine is itself a value type; it will always get boxed when you treat its MoveNext method as an Action delegate. And in this case, boxing the state machine struct also copied the async method builder struct.