Can you Yield and Resume Luajit coroutines from anywhere in C?

Question

I am trying to come up with a solution for yielding a Luajit coroutine from a C function that immediately creates a tasklet to be processed on another OS thread.

According to various Lua documentations, and things began to heavily contradict each other, this isn't entirely possible? The documentations are not very clear, and they don't explain the reasoning.

Lua 5.1 states that each each coroutine had a stack. However, there was only one global C stack. I am not entirely sure why that is a hindrance.

Lua 5.2 apparently fixes this with lua_pcallk and lua_yieldk. But the explanations are very confusing.

But none of these states the VM that I am using... which is LuaJIT 2.0.4 and LuaJIT 2.1.0.

A google search told me that Luajit 1.x had CoCo implemented, which apparently used true C stacks for each lua thread (coroutine). Which allows yielding from anywhere.

And only one search lead me to see that apparently LuaJIT 2.x does not implement coco, as each coroutine uses a C stack.

Could anyone please tell me what the problem is for yielding a coroutine from C? And verify whether or not I can safely yield/resume a luajit 2.x coroutine from c?

Answer 1

In the reference Lua implementation, each Lua coroutine has its own Lua stack, which is just an array inside of the lua_State and has no relation to the C stack. Lua is unable to save the C stack (because that's impossible in standard C), so it is unable to yield a coroutine if a C function is currently executing.

For example, if you have Lua function a calling C function b calling Lua function c, and c tries to yield, Lua will be unable to save the local variables for b (since it's a C function) and will fail.

This also applies to a lot of built-in Lua functions. As you alluded to, in Lua 5.1, the implementation did not support yielding across pcall, until Lua 5.2 apparently added special functions to make it work.

Coco is a patch to the standard Lua implementation that implements separate C stacks in coroutines, so that Lua can now "save" the C function variables. Apparently LuaJit 1.x also includes it. It won't apply to LuaJit 2.x because it's a completely different implementation of Lua.

---

LuaJit 2.x has the following paragraph in the Extensions page:

Fully Resumable VM

The LuaJIT VM is fully resumable. This means you can yield from a coroutine even across contexts, where this would not possible with the standard Lua 5.1 VM: e.g. you can yield across pcall() and xpcall(), across iterators and across metamethods.

So apparently yielding across built-in functions should work, though it's still ambiguous if it applies to arbitrary Lua C API functions. It's easy to test however; write a simple C API function that takes a Lua function and calls it, then pass it a function that yields. If it doesn't work, it should throw an error.

Note that plain C functions loaded with the FFI are not allowed to touch the Lua state at all. This includes trying to yield.