How to embed a Lua script within a C binary?

Question

I've been getting spoiled in the shell world where I can do:

./lua <<EOF
> x="hello world"
> print (x)
> EOF
hello world

Now I'm trying to include a Lua script within a C application that I expect will grow with time. I've started with a simple:

const char *lua_script="x=\"hello world\"\n"
  "print(x)\n";
luaL_loadstring(L, lua_script);
lua_pcall(L, 0, 0, 0);

But that has several drawbacks. Primarily, I have to escape the line feeds and quotes. But now I'm hitting the string length ‘1234’ is greater than the length ‘509’ ISO C90 compilers are required to support warning while compiling with gcc and I'd like to keep this program not only self-contained but portable to other compilers.

What is the best way to include a large Lua script inside of a C program, and not shipped as a separate file to the end user? Ideally, I'd like to move the script into a separate *.lua file to simplify testing and change control, and have that file somehow compiled into the executable.

Answer 1

On systems which support binutils, you can also 'compile' a Lua file into a .o with 'ld -r', link the .o into a shared object, and then link your application to the shared library. At runtime, you dlsym(RTLD_DEFAULT,...) in the lua text and can then evaluate it as you like.

To create some_stuff.o from some_stuff.lua:

ld -s -r -o some_stuff.o -b binary some_stuff.lua
objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents some_stuff.o some_stuff.o

This will get you an object file with symbols that delimit the start, end, and size of your lua data. These symbols are, as far as I know, determined by ld from the filename. You don't have control over the names, but they are consistently derived. You will get something like:

$ nm some_stuff.o 
000000000000891d R _binary_some_stuff_lua_end
000000000000891d A _binary_some_stuff_lua_size
0000000000000000 R _binary_some_stuff_lua_start

Now link some_stuff.o into a shared object like any other object file. Then, within your app, write a function that will take the name "some_stuff_lua", and do the appropriate dlsym magic. Something like the following C++, which assumes you have a wrapper around lua_State called SomeLuaStateWrapper:

void SomeLuaStateWrapper::loadEmbedded(const std::string& embeddingName)
{
    const std::string prefix = "_binary_";
    const std::string data_start = prefix + embeddingName + "_start";
    const std::string data_end = prefix + embeddingName + "_end";

    const char* const data_start_addr = reinterpret_cast<const char*>(
        dlsym(RTLD_DEFAULT, data_start.c_str()));

    const char* const data_end_addr = reinterpret_cast<const char*>(
        dlsym(RTLD_DEFAULT, data_end.c_str()));

    THROW_ASSERT(
        data_start_addr && data_end_addr,
        "Couldn't obtain addresses for start/end symbols " <<
        data_start << " and " << data_end << " for embedding " << embeddingName);

    const ptrdiff_t delta = data_end_addr - data_start_addr;

    THROW_ASSERT(
        delta > 0,
        "Non-positive offset between lua start/end symbols " <<
        data_start << " and " << data_end << " for embedding " << embeddingName);

    // NOTE: You should also load the size and verify it matches.

    static const ssize_t kMaxLuaEmbeddingSize = 16 * 1024 * 1024;
    THROW_ASSERT(
        delta <= kMaxLuaEmbeddingSize,
        "Embedded lua chunk exceeds upper bound of " << kMaxLuaEmbeddingSize << " bytes");

    namespace io = boost::iostreams;
    io::stream_buffer<io::array_source> buf(data_start_addr, data_end_addr);
    std::istream stream(&buf);

    // Call the code that knows how to feed a
    // std::istream to lua_load with the current lua_State.
    // If you need details on how to do that, leave a comment
    // and I'll post additional details.
    load(stream, embeddingName.c_str());
}

So, now within your application, assuming you have linked or dlopen'ed the library containing some_stuff.o, you can just say:

SomeLuaStateWrapper wrapper;
wrapper.loadEmbedded("some_stuff_lua");

and the original contents of some_stuff.lua will have been lua_load'ed in the context of 'wrapper'.

If, in addition, you want the shared library containing some_stuff.lua to be able to be loaded from Lua with 'require', simply give the same library that contains some_stuff.o a luaopen entry point in some other C/C++ file:

extern "C" {

int luaopen_some_stuff(lua_State* L)
{
    SomeLuaStateWrapper wrapper(L);
    wrapper.loadEmbedded("some_stuff_lua");
    return 1;
}

} // extern "C"

Your embedded Lua is now available via require as well. This works particularly well with luabind.

With SCons, it is fairly easy to educate the build system that when it sees a .lua file in the sources section of a SharedLibrary that it should 'compile' the file with the ld/objcopy steps above:

# NOTE: The 'cd'ing is annoying, but unavoidable, since
# ld in '-b binary' mode uses the name of the input file to
# set the symbol names, and if there is path info on the
# filename that ends up as part of the symbol name, which is
# no good. So we have to cd into the source directory so we
# can use the unqualified name of the source file. We need to
# abspath $TARGET since it might be a relative path, which
# would be invalid after the cd.

env['SHDATAOBJCOM'] = 'cd $$(dirname $SOURCE) && ld -s -r -o $TARGET.abspath -b binary $$(basename 
$SOURCE)'
env['SHDATAOBJROCOM'] = 'objcopy --rename-section .data=.rodata,alloc,load,readonly,data,contents $
TARGET $TARGET'

env['BUILDERS']['SharedLibrary'].add_src_builder(
    SCons.Script.Builder(
        action = [
            SCons.Action.Action(
                "$SHDATAOBJCOM",
                "$SHDATAOBJCOMSTR"
                ),
                SCons.Action.Action(
                "$SHDATAOBJROCOM",
                "$SHDATAOBJROCOMSTR"
                ),
            ],
            suffix = '$SHOBJSUFFIX',
            src_suffix='.lua',
            emitter = SCons.Defaults.SharedObjectEmitter))

I'm sure it is possible to do something like this with other modern build systems like CMake as well.

This technique is of course not limited to Lua, but can be used to embed just about any resource in a binary.