Managing stack with Lua and C++

Question

I need to pass a lua script a single string (path to file), and return 0 to many strings.

int error = 0;
lua_State *L = lua_open();
luaL_openlibs(L);

std::vector<string> list_strings;

is used to push the string onto the stack, before I load and call the source file

if ((error = luaL_loadfile(L, "src/test.lua")) == 0)
{
    lua_pushstring(L, path.c_str());

    if ((error = lua_pcall(L, 1, LUA_MULTRET, 0)) == 0)
    {
        lua_gettable(L, LUA_GLOBALSINDEX);
        lua_pcall(L,1,1,0);

        if (lua_gettop(L) == 1 && lua_istable(L,1))
        {
            int len = lua_objlen(L,1);
            for (int i=1;i =< len; i++)
            {
                lua_pushinteger(L,i);
                lua_gettable(L,1);

                const char *s = lua_tostring(L,-1);
                if (s)
                {
                    list_strings.push_back(s);
                }

                lua_pop(L,1);
            }
        }
    }
}

As it stands, I've just been copying code from examples so I'm not really sure if what I'm doing is what I want to do... I want push the path onto the stack, and call a lua function which takes that value off of the stack, and will parse the file which associates to that path.

After parsing, it should return a table containing the strings that are inside of it (you can just think of it as a function that searches for a specific string, I suppose)

edit: made more clear.

Any advice/resources? Any similar questions here? or any helpful resources?

Answer 1

I want to make sure I understand what you're doing before we see where you seem to be going wrong. You have a Lua script file. You want to execute this script, passing it a single string argument. It will do some stuff, then return zero or more strings as return values. And you want to fetch these values in your code.

OK, let's start from the top:

if ((error = lua_pcall(L, 1, LUA_MULTRET, 0)) == 0)

Normally, when you perform a lua_pcall, the third parameter tells Lua exactly how many return values are expected. If the function being called returns more than this number, those return values are discarded. If it returns fewer than this number, then additional NIL values are used to fill out the count.

LUA_MULTRET tells Lua not to do this. When this is used, all results are pushed onto the stack.

Now, since you neglected to post your script, I have to make some guesses as to what your script looks like. You are returning multiple strings, but you never say how this happens. Lua, as a language, allows multiple return values:

return "string1", "string2";

This results in 2 strings being pushed onto the stack. This is different from:

return {"string1", "string2"};

This puts one object onto the stack: a table. The table contains 2 strings. See the difference?

Looking at your code, it seems like you expect the Lua script to return a table of strings, not multiple return values.

In which case, you should call your Lua script like this:

if ((error = lua_pcall(L, 1, 1, 0)) == 0)

This tells Lua that you expect a single return value, and if the user doesn't provide one, Lua will push a NIL onto the stack.

Now let's talk about the stack. The state of the stack was this before issuing the function call:

2- {string: path.c_str()}
1- {function: loaded from file "src/test.lua"}

This is from the top of the stack to the "bottom". If you use the lua_pcall that I gave you, you will get the following on your stack:

1- {return value}

The lua_pcall will remove the argument(s) and function from the stack. So it will pop N + 1 items from the stack, where N is the number of arguments to the Lua function as specified by lua_pcall (the second parameter). Therefore, Lua will pop 2 things off the stack. It will then push exactly 1 value onto the stack: the return value (or NIL if there was no return value).

So that gets us past the function call. If all has gone well, we now expect the stack to contain:

1- {table: returned from function}

However, all may not have gone well. The script may have returned NIL. Or something else; there's no guarantee that it was a table. So, the next step is to verify the return value (note: this is where your code stops making sense, so this is all new).

if(lua_istable(L, -1))

lua_istable does exactly what the name suggests: determine if the given item is a table. But what does that "-1" mean, and why isn't it the "1" you had in your code?

This argument is a reference to a location on the stack. Lua's stack is also Lua's register file. This means that, unlike a real stack, you are allowed to peak at any element on the stack. Elements on the stack have an absolute location on the stack. Now, here's what our stack looks like again:

1- {return value}

That "1" I wrote is the absolute location on the stack of this value. I can push values and pop values, but unless I pop this value, it's location will always be "1".

However, it is only "1" because our stack started out empty. It's somewhat rude to assume this (as it can really bite you if the stack isn't empty. The Lua docs do helpfully state when you can assume the stack really is empty, or if its not, what is already on the stack). Therefore, you can use relative locations.

And that's what "-1" is: it is the first stack index from the top of the stack. Our lua_pcall function as defined above will pop 2 items from the stack (the argument and the function), and push 1 item (the return value or NIL). Therefore, "-1" will always refer to our return value.

Thus, we check to see if the stack index "-1" (the top of the stack) is a table. If it isn't, then fail. If it is, then we can parse our list.

And this is where we get to list parsing. The first step is to get the number of items in the list:

int len = lua_objlen(L, -1);
list_strings.reserve(len);

The second one is just a nicety, so that you're not allocating a bunch of times. You know exactly how many strings are going to be in that list, so you may as well let the list know ahead of time, right?

lua_objlen gets the number of array elements in the table. Note that this can return zero, but our loop will handle that case.

Next, we walk the table, pulling out the strings.

for (int i=0; i < len; i++) {
    //Stuff from below.
}

Remember that Lua uses 1-base indices. I personally prefer using 0-base indices while in C/C++ code, even code that interfaces with Lua. So I do the translation as late as possible. But you don't have to.

Now, for the contents of the loop. The first step is to get the table entry from the table. To do that, we need to give Lua an index and tell Lua to get that index from the table:

lua_pushinteger(L, i + 1);
lua_gettable(L, -2);

Now, the first function pushes the index onto the stack. After that, our stack looks like this:

2- {integer: i + 1}
1- {table: returned from function}

The lua_gettable function deserves more explanation. It takes a key (remember: table keys in Lua do not have to be integers) and a table, and returns the value associated with that key in that table. Or NIL, if no value is associated there. But the way it works is a bit odd.

It assumes that the top of the stack is the key. So the parameter it takes is the stack location of the table that the key will index into. We use "-2" because, well, look at the stack. The table is 2 from the top since we pushed an integer; therefore we use "-2".

After this, our stack looks like this:

2- {value: from table[i + 1]}
1- {table: returned from function}

Now that we have gotten a value, we must verify that it is a string, and then get its value.

size_t strLen = 0;
const char *theString = lua_tolstring(L, -1, &strLen);

This function does all of these at once. If the value we got from the table is not a string (or a number, since Lua will auto-convert numbers to strings), then theString will be NULL. Otherwise, theString will have a Lua-owned pointer (do not delete) to the string. strLen will also have the length of the string.

Quick aside: Lua strings are NULL-terminated, but they also can internally contain NULL characters. C-strings aren't allowed to do this, but C++ std::strings are. That's why I don't use lua_tostring the way you did; C++ strings can store Lua strings exactly as they are.

Now that we have the string data from Lua, we need to put it into our list. To avoid unnecessary copies, I prefer this syntax:

list_strings.push_back();
list_strings.back().assign(theString, strLen);

If I were using a C++11-enabled standard library and compiler, I would have just used list_strings.emplace_back(theString, strLen);, relying on the emplace_back function to construc the std::string in-place. This neatly avoids making more copies of the string than necessary.

There is one final bit of cleanup we need to do. Our stack still has two values on it: the string and the table. We're done with the string, so we need to get rid of it. This is done by poping one entry from the Lua stack:

lua_pop(L, 1);

Here, the "1" is the number of entries to pop, not the stack location.

Do you understand how stack management works in Lua now?

---

1) Looking at the state of the stack before the call... luaL_loadfile pushes a function to the stack? Or does lua_pcall?

Assuming you haven't done anything with the Lua state besides create it, then stack is empty before luaL_loadfile. And yes, luaL_loadfile pushes a function onto the stack. This function represents the file that was loaded.

3) What would the result of the stack be, if after making the function call it returned an error value?

Exactly what the documentation says. Now that you understand how the stack works, you should read through the docs. The Programming in Lua book is also recommended. Version 5.0 is available online for free, but the 5.1 book costs money. The 5.0 book is still a useful starting point.

4) list_strings.reserve(len); As for this... This lua script is actually embedded in a small C program that recurses through a code base and will collect ALL of the strings that the lua script returns from ALL of the files... I don't know exactly how reserve works, but What I'm saying is that I will be using many tables to add strings to this list... Should reserve just be not used in that case? or still used...

std::vector::reserve ensures that the std::vector will contain at least enough space for X elements, where X is the value you pass it. I did this because Lua tells you how many elements are in the table, so there is no need to let the std::vector expand on its own. You can make it do one memory allocation for everything, rather than letting the std::vector::push_back function allocate more memory as needed.

This is useful so long as you call your Lua script once. That is, it gets a single return value from Lua. No matter how large the table returned is, this will work. If you call your Lua script (from C++) multiple times, then there's no way to know ahead of time how much memory to reserve. You could reserve space for each table you get back, but it's possible for the std::vector's default allocation scheme to beat you in number of allocations for large datasets. So in that case, I wouldn't bother with reserve.

However, it wouldn't be unwise to start off with a healthy-sized reserve, as kind of a default case. Pick a number that you think would be "big enough", and reserve that much space.