Why does `for key, value in table` stop working in Lua 5.1 and later?

Question

The source code for every Lua release is available in one handy package, which can be extracted and the versions can be compiled all at once:

wget https://www.lua.org/ftp/lua-all.tar.gz
tar xvf lua-all.tar.gz --strip-components=1
make

Afterwards, you can run example code in all versions to compare:

cat <<EOF >pairs.lua
for key, value in pairs({[2] = "a", [1] = "b", ["c"] = "d"}) do
    print(key .. " -> " .. value)
end
EOF
ls -1d */ | sed 's/\/$//' | sort | while IFS= read -r f; do printf %s\\n "" "$f"; "$f/lua" pairs.lua; done

Outputs:

lua-1.0
lua: syntax error near "key" at line 1 in file "pairs.lua"

lua-1.1
lua: syntax error near "key" at line 1 in file "pairs.lua"

lua-2.1
lua: syntax error near "key" at line 1 in file "pairs.lua"

lua-2.2
lua: syntax error near "key" at line 1 in file `pairs.lua'
Active Stack:
    fallback error

lua-2.4
lua: syntax error; last token read: "key" at line 1 in file `pairs.lua'
Active Stack:
    `error' fallback
lua: error trying to run file pairs.lua

lua-2.5
lua: syntax error;
> last token read: "key" at line 1 in file pairs.lua
Active Stack:
    `error' fallback

lua-2.5.1
lua: syntax error;
> last token read: "key" at line 1 in file pairs.lua
Active Stack:
    `error' fallback

lua-3.0
lua: syntax error;
> last token read: "key" at line 1 in file pairs.lua

lua-3.1
lua: `=' expected;
  last token read: `key' at line 1 in chunk `pairs.lua'

lua-3.2

lua-3.2.1
lua error: `=' expected;
  last token read: `key' at line 1 in file `pairs.lua'

lua-3.2.2
lua error: `=' expected;
  last token read: `key' at line 1 in file `pairs.lua'

lua-4.0
error: attempt to call global `pairs' (a nil value)
stack traceback:
   1:  main of file `pairs.lua' at line 1

lua-4.0.1
error: attempt to call global `pairs' (a nil value)
stack traceback:
   1:  main of file `pairs.lua' at line 1

lua-5.0
1 -> b
2 -> a
c -> d

lua-5.0.1
1 -> b
2 -> a
c -> d

lua-5.0.2
1 -> b
2 -> a
c -> d

lua-5.0.3
1 -> b
2 -> a
c -> d

lua-5.1
2 -> a
1 -> b
c -> d

lua-5.1.1
2 -> a
1 -> b
c -> d

lua-5.1.2
2 -> a
1 -> b
c -> d

lua-5.1.3
1 -> b
2 -> a
c -> d

lua-5.1.4
1 -> b
2 -> a
c -> d

lua-5.1.5
1 -> b
2 -> a
c -> d

lua-5.2.0
2 -> a
1 -> b
c -> d

lua-5.2.1
2 -> a
1 -> b
c -> d

lua-5.2.2
2 -> a
1 -> b
c -> d

lua-5.2.3
2 -> a
1 -> b
c -> d

lua-5.2.4
2 -> a
1 -> b
c -> d

lua-5.3.0
1 -> b
2 -> a
c -> d

lua-5.3.1
1 -> b
2 -> a
c -> d

lua-5.3.2
1 -> b
2 -> a
c -> d

lua-5.3.3
1 -> b
2 -> a
c -> d

lua-5.3.4
1 -> b
2 -> a
c -> d

lua-5.3.5
1 -> b
2 -> a
c -> d

lua-5.3.6
1 -> b
2 -> a
c -> d

lua-5.4.0
1 -> b
2 -> a
c -> d

lua-5.4.1
1 -> b
2 -> a
c -> d

lua-5.4.2
1 -> b
2 -> a
c -> d

lua-5.4.3
1 -> b
2 -> a
c -> d

lua-5.4.4
1 -> b
2 -> a
c -> d

lua-5.4.5
1 -> b
2 -> a
c -> d

lua-5.4.6
1 -> b
2 -> a
c -> d

lua-5.4.7
1 -> b
2 -> a
c -> d

And:

cat <<EOF >ipairs.lua
for key, value in ipairs({[2] = "a", [1] = "b", ["c"] = "d"}) do
    print(key .. " -> " .. value)
end
EOF
ls -1d */ | sed 's/\/$//' | sort | while IFS= read -r f; do printf %s\\n "" "$f"; "$f/lua" ipairs.lua; done

Outputs:

lua-1.0
lua: syntax error near "key" at line 1 in file "ipairs.lua"

lua-1.1
lua: syntax error near "key" at line 1 in file "ipairs.lua"

lua-2.1
lua: syntax error near "key" at line 1 in file "ipairs.lua"

lua-2.2
lua: syntax error near "key" at line 1 in file `ipairs.lua'
Active Stack:
    fallback error

lua-2.4
lua: syntax error; last token read: "key" at line 1 in file `ipairs.lua'
Active Stack:
    `error' fallback
lua: error trying to run file ipairs.lua

lua-2.5
lua: syntax error;
> last token read: "key" at line 1 in file ipairs.lua
Active Stack:
    `error' fallback

lua-2.5.1
lua: syntax error;
> last token read: "key" at line 1 in file ipairs.lua
Active Stack:
    `error' fallback

lua-3.0
lua: syntax error;
> last token read: "key" at line 1 in file ipairs.lua

lua-3.1
lua: `=' expected;
  last token read: `key' at line 1 in chunk `ipairs.lua'

lua-3.2

lua-3.2.1
lua error: `=' expected;
  last token read: `key' at line 1 in file `ipairs.lua'

lua-3.2.2
lua error: `=' expected;
  last token read: `key' at line 1 in file `ipairs.lua'

lua-4.0
error: attempt to call global `ipairs' (a nil value)
stack traceback:
   1:  main of file `ipairs.lua' at line 1

lua-4.0.1
error: attempt to call global `ipairs' (a nil value)
stack traceback:
   1:  main of file `ipairs.lua' at line 1

lua-5.0
1 -> b
2 -> a

lua-5.0.1
1 -> b
2 -> a

lua-5.0.2
1 -> b
2 -> a

lua-5.0.3
1 -> b
2 -> a

lua-5.1
1 -> b
2 -> a

lua-5.1.1
1 -> b
2 -> a

lua-5.1.2
1 -> b
2 -> a

lua-5.1.3
1 -> b
2 -> a

lua-5.1.4
1 -> b
2 -> a

lua-5.1.5
1 -> b
2 -> a

lua-5.2.0
1 -> b
2 -> a

lua-5.2.1
1 -> b
2 -> a

lua-5.2.2
1 -> b
2 -> a

lua-5.2.3
1 -> b
2 -> a

lua-5.2.4
1 -> b
2 -> a

lua-5.3.0
1 -> b
2 -> a

lua-5.3.1
1 -> b
2 -> a

lua-5.3.2
1 -> b
2 -> a

lua-5.3.3
1 -> b
2 -> a

lua-5.3.4
1 -> b
2 -> a

lua-5.3.5
1 -> b
2 -> a

lua-5.3.6
1 -> b
2 -> a

lua-5.4.0
1 -> b
2 -> a

lua-5.4.1
1 -> b
2 -> a

lua-5.4.2
1 -> b
2 -> a

lua-5.4.3
1 -> b
2 -> a

lua-5.4.4
1 -> b
2 -> a

lua-5.4.5
1 -> b
2 -> a

lua-5.4.6
1 -> b
2 -> a

lua-5.4.7
1 -> b
2 -> a

As expected, the code starts working with Lua 5.0, which is when pairs and ipairs was introduced.

My question is about the following example:

cat <<EOF >neither.lua
for key, value in {[2] = "a", [1] = "b", ["c"] = "d"} do
    print(key .. " -> " .. value)
end
EOF
ls -1d */ | sed 's/\/$//' | sort | while IFS= read -r f; do printf %s\\n "" "$f"; "$f/lua" neither.lua; done

Outputs:

lua-1.0
lua: syntax error near "key" at line 1 in file "neither.lua"

lua-1.1
lua: syntax error near "key" at line 1 in file "neither.lua"

lua-2.1
lua: syntax error near "key" at line 1 in file "neither.lua"

lua-2.2
lua: syntax error near "key" at line 1 in file `neither.lua'
Active Stack:
    fallback error

lua-2.4
lua: syntax error; last token read: "key" at line 1 in file `neither.lua'
Active Stack:
    `error' fallback
lua: error trying to run file neither.lua

lua-2.5
lua: syntax error;
> last token read: "key" at line 1 in file neither.lua
Active Stack:
    `error' fallback

lua-2.5.1
lua: syntax error;
> last token read: "key" at line 1 in file neither.lua
Active Stack:
    `error' fallback

lua-3.0
lua: syntax error;
> last token read: "key" at line 1 in file neither.lua

lua-3.1
lua: `=' expected;
  last token read: `key' at line 1 in chunk `neither.lua'

lua-3.2

lua-3.2.1
lua error: `=' expected;
  last token read: `key' at line 1 in file `neither.lua'

lua-3.2.2
lua error: `=' expected;
  last token read: `key' at line 1 in file `neither.lua'

lua-4.0
1 -> b
c -> d
2 -> a

lua-4.0.1
1 -> b
c -> d
2 -> a

lua-5.0
1 -> b
2 -> a
c -> d

lua-5.0.1
1 -> b
2 -> a
c -> d

lua-5.0.2
1 -> b
2 -> a
c -> d

lua-5.0.3
1 -> b
2 -> a
c -> d

lua-5.1
lua-5.1/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.1.1
lua-5.1.1/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.1.2
lua-5.1.2/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.1.3
lua-5.1.3/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.1.4
lua-5.1.4/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.1.5
lua-5.1.5/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: ?

lua-5.2.0
lua-5.2.0/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.2.1
lua-5.2.1/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.2.2
lua-5.2.2/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.2.3
lua-5.2.3/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.2.4
lua-5.2.4/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.0
lua-5.3.0/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.1
lua-5.3.1/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.2
lua-5.3.2/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.3
lua-5.3.3/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.4
lua-5.3.4/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.5
lua-5.3.5/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.3.6
lua-5.3.6/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.0
lua-5.4.0/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.1
lua-5.4.1/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.2
lua-5.4.2/lua: neither.lua:1: attempt to call a table value
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.3
lua-5.4.3/lua: neither.lua:1: for iterator 'for iterator' is not callable (a table value)
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.4
lua-5.4.4/lua: neither.lua:1: attempt to call a table value (for iterator 'for iterator')
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.5
lua-5.4.5/lua: neither.lua:1: attempt to call a table value (for iterator 'for iterator')
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.6
lua-5.4.6/lua: neither.lua:1: attempt to call a table value (for iterator 'for iterator')
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

lua-5.4.7
lua-5.4.7/lua: neither.lua:1: attempt to call a table value (for iterator 'for iterator')
stack traceback:
    neither.lua:1: in main chunk
    [C]: in ?

This for key, value in table syntax starts working with Lua 4.0, but stops working in Lua 5.1 and later. What is the reason for this? While it was working, was it equivalent to pairs?

There's already quite a few posts about generic for loops in Lua - like How to iterate through table in Lua?, What is the difference between pairs() and ipairs() in Lua?, Is there any way to loop through an array without using pairs()? or for in loop with key value pairs, to mention a few - but as far as I can tell, none of them contain an answer to this question.

Answer 1

Lua 5.0 introduced the more powerful generic for statement which works over functions and can be used to implement generic iterators. Lua 5.0 kept "for k,v in t" for compatibility but marked it deprecated. As such, it was removed in Lua 5.1. See the section "Incompatibilities with version 4.0" in the Lua 5.0 Reference Manual.

In Lua 5.0+, you can use "for k,v in next,t", which is equivalent to "for k,v in pairs(t)".

Answer 2

With the help of the other answer, I was able to investigate the changes around for loops that happened in Lua from 4.0 to 5.1 and note down my findings. Initially, in Lua 4.0, there are two forms of for loops: "numical for" and "table for". The latter matches this question's for key, value in table and is described in https://www.lua.org/manual/4.0/manual.html#4.4 as:

The table for statement traverses all pairs (index,value) of a given table. It has the following syntax:

stat ::= for name `,' name in exp1 do block end

A for statement like

for index, value in exp do block end

is equivalent to the code:

do
  local _t = exp
  local index, value = next(t, nil)
  while index do
    block
    index, value = next(t, index)
  end
end

Note the following:

• _t is an invisible variable. The name is here for explanatory purposes only.
• The behavior is undefined if you assign to index inside the block.
• The behavior is undefined if you change the table _t during the traversal.
• The variables index and value are local to the statement; you cannot use their values after the for ends.
• You can use break to exit a for. If you need the value of index or value, assign them to other variables before breaking.
• The order that table elements are traversed is undefined, even for numerical indices. If you want to traverse indices in numerical order, use a numerical for.

This means that for key, value in table is a special form of the for loop, explicitly for tables, called "table for", introduced with Lua 4.0. In Lua 5.0, another, more powerful form of for loop was introduced: the "generic for". It has a different syntax in that it expects an iterator instead of simply a table. (Refer to "generic for" in the Lua 5.0 manual for more information about iterators.)

As to what happened to "table for", chapter "Incompatibilities with Previous Versions" in https://www.lua.org/manual/5.0/manual.html mentions:

The old construction for k,v in t, where t is a table, is deprecated (although it is still supported). Use for k,v in pairs(t) instead.

So it was deprecated in 5.0 already, but wasn't removed until 5.1 - which, unfortunately, doesn't seem to be to mentioned in the 5.1 manual.

Finally, according to the code excerpt from the 4.0 manual above, "table for" like for key, value in table did, while it existed, behave like "generic for" with pairs like for key, value in pairs(table) in Lua 5.0 and later.