How do I find an `f` that makes `pcall(pcall, f)` return `false` in Lua?

Question

I'm using Lua 5.3.

Reading 'Programming in Lua 3rd edition', I came into an exercise asking me to find an f that makes pcall(pcall, f) return false. I think this is equivalent to make pcall(f) raise an error. But seems use pcall on anything won't do that. For example, let f = nil and nothing bad happens.

However, I found on the internet that letting f = (function() end)() does the trick. But I don't understand why. Here f is just the return value (which is nil) of the function function() end right?

Answer 1

This does not work

> f = (function() end)()
> pcall(pcall, f)
true    false   attempt to call a nil value

but this does:

> pcall(pcall,(function() end)())
false   bad argument #1 to 'pcall' (value expected)

The difference is that (function() end)() returns no value, which is different from returning nil. Functions written in Lua cannot make this distinction but pcall is written in C and can make this distinction. (Other examples are print and type.)

Note that this behaves as expected (equivalent to the first attempt):

> pcall(pcall,(function() return nil end)())
true    false   attempt to call a nil value

Answer 2

This is because the code you are calling is equivalent to pcall(pcall), not to pcall(pcall, nil). The function doesn't return anything; it's adjusted to nil in some contexts, but is still different from nil in other contexts (as is the case in this example).