Lua length operator (#) with nil values

Question

After reading this topic and after experimenting a bit, I am trying to understand how the Lua length operator works when a table contains nil values.

Before I started to investigate, I thought that the length was simply the number of consecutive non-nil elements, starting at index 1:

print(#{nil})         -- 0
print(#{"o"})         -- 1
print(#{"o",nil})     -- 1
print(#{"o","o"})     -- 2
print(#{"o","o",nil}) -- 2

That looks pretty simple, right?

But my headache started when I accidentally added an element after a nil-terminated table:

print(#{"o",nil,"o"})

My guess was that it should probably print 1 because it would stop counting when the first nil is found. Or maybe it should print 2 if the length operator is greedy enough to look for non-nil elements after the first nil. But the above code prints 3.

So I’ve ran several other tests to see what happens:

-- nil before the end
print(#{nil,"o"})     -- 2
print(#{nil,"o","o"}) -- 3
print(#{"o",nil,"o"}) -- 3

-- several nil elements
print(#{"o",nil,nil}) -- 1
print(#{nil,"o",nil}) -- 0
print(#{nil,nil,"o"}) -- 3

I should mention that repl.it currently uses Lua 5.1.5 which is rather old, but if you test with the Lua demo, which currently uses Lua 5.3.5, you’ll get the same results.

By looking at those results and by looking at this answer, I assume that:

• if the last element is not nil, the length operator returns the full size of the table, including nil entries if any
• if the last element is nil, it counts the number of consecutive non-nil and stops counting at the first nil

Are those assumptions correct?

Can we predict a 100% well-defined behavior when a table contains one or several nil values?

The Lua documentation states that the length of a table is only defined if the table is a sequence. Does that mean that the length operator has undefined behavior for non-sequences?

Apart from the length operator, can nil values cause any trouble in a table?

Answer 1

We can predict some behaviour, but it is not standardised, and as such you should never rely on it. It's quite possible that the behaviour may change within this major version of Lua.

Should you ever need to fill a table with nil values, I suggest wrapping the table and replace holes with a unique placeholder value (eg. NIL={}; if v==nil then t[k]=NIL end, this is quite cheap to test against and safe.).

That said...

As there is even a difference in the result of # depending on how the table is defined, you'll have to distinguish between statically defined (constant) tables and dynamic defined (muted) tables.

Static table definitions:

#{nil,nil,nil,nil,nil,  1} -- 6
#{3, 2, nil, 1} -- 4

#{nil,nil,nil,  1,  1,nil} -- 0
#{nil,nil,  1,  1,  1,nil} -- 5
#{nil,  1,  1,  1,  1,nil} -- 5
#{nil,nil,nil,nil,  1,nil} -- 0
#{nil,nil,  1,nil,  1,nil,nil} -- 5
#{nil,nil,nil,  1,nil,nil,  1,nil} -- 4

Using this kind of definition, as long as the last value is non-nil, you will get a length equal to the position of the last value. If the last value is nil, Lua starts a (non-linear) search from the tail until it finds the first non-nil value.

Dynamic data definition

local x={}; x[5]=1;print(#x) -- 0
local x={}; x[1]=1;x[2]=1;x[3]=1;x[5]=1;print(#x) -- 3
local x={}; x[1]=1;x[2]=1;x[4]=1;x[5]=1;print(#x) -- 5

#{[5]=1} -- 0
local x={nil,nil,nil,1};x[5]=1;print(#x) -- 0

As soon as the table was changed once, the operator works the other way (that includes static definitions with []). If the first element is nil, # always returns 0, but if not it starts a search that I did not investigate further (I guess you can check the sources, though I don't think it's a standard binary search), until it finds a nil value that is preceded by a non-nil value.

As said before, relying on this behaviour is not a good idea, and invites lots of issues down the road. Though if you want to make a nasty unmaintainable program to mess with a colleague, that's a sure way to do it.