How to implement a Lua container (virtual file system) module loader in C#

Question

Sounds a little bit scary isn't it?

Some background information, I want to load a tar archive which contains some lua modules into my C# application using LuaInterface. The easiest way would be to extract these files to a temp folder, modify the lua module search path and read them with require as usual. But I do not want to put these scripts somewhere on the file system.

So I thought it should be possible to load the tar-archive with the #ziplib I know there are a lot of lua implementations for tar and stuff like that. But the #zlib is already part of the project.

After successfully loading the file as strings(streams) out of the archive I should be able to pass them into lua.DoString(...) in C# via LuaInterface.

But simply loading modules by a dostring or dofile does not work if modules have a line like this: "module(..., package.seeall)" There is a error reportet like passing argument 1 a nil, but string expected.

The other problem is a module may depend on other modules which are also located in the tar archive.

One possible solution should be to define a custom loader as described here.

My idea is to implement such a loader in C# with the #ziplib and map this loader into the lua stack of my C# application.

Does anyone of you had a similar task to this? Are there any ready to use solutions which already address problems like this?

The tar file is not must have but a nice to have package format.

Is this idea feasible or totally unfeasible?

I've written some example class to extract the lua files from the archive. This method works as loader and return a lua function.

namespace LuaInterfaceTest
{
 class LuaTarModuleLoader
 {
    private LuaTarModuleLoader() { }
    ~LuaTarModuleLoader()
    {
        in_stream_.Close();
    }
    public LuaTarModuleLoader(Stream in_stream,Lua lua )
    {
        in_stream_ = in_stream;
        lua_ = lua;
    }

    public LuaFunction load(string modulename, out string error_message)
    {
        string lua_chunk = "test=hello";
        string filename = modulename + ".lua";
        error_message = "Unable to locate the file";
        in_stream_.Position = 0; // rewind
        Stream gzipStream = new BZip2InputStream(in_stream_);
        TarInputStream tar = new TarInputStream(gzipStream);
        TarEntry tarEntry;
        LuaFunction func = null;
        while ((tarEntry = tar.GetNextEntry()) != null)
        {
            if (tarEntry.IsDirectory)
            {
                continue;
            }
            if (filename == tarEntry.Name)
            {
                MemoryStream out_stream = new MemoryStream();
                tar.CopyEntryContents(out_stream);
                out_stream.Position = 0; // rewind
                StreamReader stream_reader = new StreamReader(out_stream);
                lua_chunk = stream_reader.ReadToEnd();
                func = lua_.LoadString(lua_chunk, filename);
                string dum = func.ToString();
                error_message = "No Error!";
                break;
            }
        }
        return func;
    }
    private Stream in_stream_;
    private Lua lua_;
}

}

I try to register the load method like this in the LuaInterface

        Lua lua = new Lua();
        GC.Collect();
        Stream inStream = File.OpenRead("c:\\tmp\\lua_scripts.tar.bz2");
        LuaTarModuleLoader tar_loader = new LuaTarModuleLoader(inStream, lua);
        lua.DoString("require 'CLRPackage'");
        lua.DoString("import \"ICSharpCode.SharpZipLib.dll\"");
        lua.DoString("import \"System\"");
        lua["container_module_loader"] = tar_loader;
        lua.DoString("table.insert(package.loaders, 2, container_module_loader.load)");
        lua.DoString("require 'def_sensor'");

If I try it this way I'll get an exception while the call to require :

"instance method 'load' requires a non null target object"

I tried to call the load method directly, here I have to use the ":" notation.

lua.DoString("container_module_loader:load('def_sensor')");

If I call the method like that I hit a breakpoint in the debugger which is place on top of the method so everything works as expected.

But If I try to register the method with ":" notation I get an exception while registering the method:

lua.DoString("table.insert(package.loaders, 2, container_module_loader:load)");

"[string "chunk"]:1: function arguments expected near ')'"

Answer 1

In LÖVE they have that working. All Lua files are inside one zip file, and they work, even if ... is used. The library they use is PhysicsFS.

Have a look at the source. Probably /modules/filesystem will get you started.

Answer 2

I finally got the trick ;-)

One Problem I currently not really understand is that my loader should not return any string. Here is my solution:

The loader Class itself:

namespace LuaInterfaceTest
{
class LuaTarModuleLoader
{
    private LuaTarModuleLoader() { }
    ~LuaTarModuleLoader()
    {
        in_stream_.Close();
    }
    public LuaTarModuleLoader(Stream in_stream,Lua lua )
    {
        in_stream_ = in_stream;
        lua_ = lua;
    }

    public LuaFunction load(string modulename)
    {
        string lua_chunk = "";
        string filename = modulename + ".lua";
        in_stream_.Position = 0; // rewind
        Stream gzipStream = new BZip2InputStream(in_stream_);
        TarInputStream tar = new TarInputStream(gzipStream);
        TarEntry tarEntry;
        LuaFunction func = null;
        while ((tarEntry = tar.GetNextEntry()) != null)
        {
            if (tarEntry.IsDirectory)
            {
                continue;
            }
            if (filename == tarEntry.Name)
            {
                MemoryStream out_stream = new MemoryStream();
                tar.CopyEntryContents(out_stream);
                out_stream.Position = 0; // rewind
                StreamReader stream_reader = new StreamReader(out_stream);
                lua_chunk = stream_reader.ReadToEnd();
                func = lua_.LoadString(lua_chunk, modulename);
                string dum = func.ToString();
                break;
            }
        }
        return func;
    }
    private Stream in_stream_;
    private Lua lua_;
}

}

And how to use the loader, I am not sure if all the package stuff is really needed. But I had to wrap up the call with ":" notation and hide it behind my "load_wrapper" function.

        string load_wrapper = "local function load_wrapper(modname)\n return container_module_loader:load(modname)\n end";
        Lua lua = new Lua();
        GC.Collect();
        Stream inStream = File.OpenRead("c:\\tmp\\lua_scripts.tar.bz2");
        LuaTarModuleLoader tar_loader = new LuaTarModuleLoader(inStream, lua);
        lua.DoString("require 'CLRPackage'");
        lua.DoString("import \"System\"");
        lua["container_module_loader"] = tar_loader;
        lua.DoString(load_wrapper);

        string loader_package = "module('my_loader', package.seeall) \n";
        loader_package += load_wrapper + "\n";
        loader_package += "table.insert(package.loaders, 2, load_wrapper)";
        lua.DoString(loader_package);
        lua.DoFile("./load_modules.lua");

I hope this may also helps some other