What does it mean for "With a stackless coroutine, only the top-level routine may be suspended."

Question

I found that statement from here. At first I was astonished because I believe that makes stackless coroutines almost useless (And C++ coroutine TS is stackless). So I wrote a demo (In visual studio using C++ coroutine TS):

#include<experimental/coroutine>
#include<iostream>
#include<thread>
#include<mutex>
#include<future>
#include<chrono>

using namespace std;
using namespace std::chrono;
using namespace std::experimental;

class AsyncQueue {
public:
    class Awaitable {
        friend AsyncQueue;
        AsyncQueue& mQueue;
        coroutine_handle<> mCoroutineHandle;
        Awaitable* mNext = nullptr;
    public:
        Awaitable(AsyncQueue& queue):mQueue(queue){}

        bool await_ready() const noexcept {
            return false;
        }

        bool await_suspend(coroutine_handle<> coroutineHandle) noexcept
        {
            mCoroutineHandle = coroutineHandle;
            mQueue.enqueue(this);
            return true;
        }

        void await_resume() noexcept {}
    };
private:
    mutex mMutex;
    Awaitable* mHead = nullptr;
    Awaitable* mTail = nullptr;
    void enqueue(Awaitable* awaitable){
        lock_guard<mutex> g{ mMutex };
        if (mTail) {
            mTail->mNext = awaitable;
            mTail = awaitable;
        }
        else {
            mTail = awaitable;
            mHead = mTail;
        }
    }

    Awaitable* dequeue() {
        lock_guard<mutex> g{ mMutex };
        Awaitable* result = mHead;
        mHead = nullptr;
        mTail = nullptr;
        return result;
    }

public:
    Awaitable operator co_await() noexcept {
        return Awaitable{ *this };
    }

    bool poll() {
        Awaitable* awaitables = dequeue();
        if (!awaitables) {
            return false;
        }
        else {
            while (awaitables) {
                awaitables->mCoroutineHandle.resume();
                awaitables = awaitables->mNext;
            }
            return true;
        }
    }
};


AsyncQueue toBackgroundThread;
AsyncQueue toMainThread;

std::future<void> secondLevel(int id)
{
    co_await toBackgroundThread;
    cout << id << " run on " << this_thread::get_id() << endl;
    co_await toMainThread;
    cout << id << " run on " << this_thread::get_id() << endl;
}

std::future<void> topLevel() {
    co_await secondLevel(1);
    co_await secondLevel(2);
}

void listen(AsyncQueue& queue) {
    while (true) {
        if (!queue.poll()) {
            this_thread::sleep_for(100ms);
        }
    }
}

int main() {
    thread([]() {
        listen(toBackgroundThread);
    }).detach();

    topLevel();

    listen(toMainThread);
}

coroutine topLevel calls two secondLevel (which I believe are suspendable non-top-level routines), and it works fine. The code above prints:

1 run on 16648
1 run on 3448
2 run on 16648
2 run on 3448

From that answer it is claimed that This prohibits providing suspend/resume operations in routines within a general-purpose library. I see no prohibitions here.

Answer 1

In each invocation of co_await, only the top level coroutine is suspended. To suspend a lower level, that level must suspend itself explicitly. And at that point, it is now the current "top level". So in every case, only the current top level gets suspended.

Compare this to a purely hypothetical stackful coroutine library:

//This function will always print the same thread ID.
void secondLevel(int id)
{
    while(!toBackgroundThread.poll())
      suspend_coroutine();

    cout << id << " run on " << this_thread::get_id() << endl;

    while(!toBackgroundThread.poll())
      suspend_coroutine();

    cout << id << " run on " << this_thread::get_id() << endl;
}

void topLevel() {
    secondLevel(1);
    secondLevel(2);
}

void listen(AsyncQueue& queue) {
    while (true) {
        if (!queue.poll()) {
            this_thread::sleep_for(100ms);
        }
    }
}

int main() {
    thread([]() {
        listen(toBackgroundThread);
    }).detach();

    auto coro = create_coroutine(topLevel);
    coro.switch_to();

    toMainThread.ready(); //Notes that the main thread is waiting
    while (true) {
        if (!toMainThread.poll()) {
            coro.switch_to();
        }
    }
};

topLevel doesn't have any explicit suspension machinery. Yet its execution suspends whenever any function it calls suspends execution. The entire call-stack, defined by the function given to create_coroutine and everything it calls, suspends. That's how a stackful coroutine works.

That is what is being contrasted with when it comes to stackless coroutines. In the stackless version, every function that needs to suspend must be specifically coded to do so. And thus isn't really "general purpose" anymore; it's now special-cased to suspending scenarios.