Lua compiled scripts on Mac OS X - Intel vs PPC

Question

Been using Lua 5.0 in a Mac OS X universal binary app for some years. Lua scripts are compiled using luac and the compiled scripts are bundled with the app. They have worked properly in Tiger and Leopard, Intel or PPC.

To avoid library problems at the time, I simply added the Lua src tree to my Xcode project and compiled as is, with no problems.

It was time to update to a more modern version of Lua so I replaced my source tree with that of 5.1.4. I rebuilt luac using make macosx (machine is running Leopard on Intel).

Uncompiled scripts work properly in Tiger and Leopard, Intel and PPC, as always.

However, now compiled scripts fail to load on PPC machines.

So I rebuilt luac with the 'ansi' flag, and recompiled my scripts. Same error. Similarly, a build flag of 'generic' produced no joy.

Can anyone please advise on what I can do next?

Answer 1

Lua's compiled scripts are pretty much the raw bytecode dumped out after a short header. The header documents some of the properties of the platform used to compile the bytecode, but the loader only verifies that the current platform has the same properties.

Unfortunately, this creates problems when loading bytecode compiled on another platform, even if compiled by the very same version of Lua. Of course, scripts compiled by different versions of Lua cannot be expected to work, and since the version number of Lua is included in the bytecode header, the attempt to load them is caught by the core.

The simple answer is to just not compile scripts. If Lua compiles the script itself, you only have to worry about possible version mismatches between Lua cores in your various builds of your application, and that isn't hard to deal with.

Actually supporting a full cross compatibility for compiled bytecode is not easy. In that email, Mike Pall identified the following issues:

1. Endianess: swap on output as needed.

2. sizeof(size_t), affects huge string constants: check for overflow when downgrading.

3. sizeof(int), affectsMAXARG_Bx and MAXARG_sBx: check for overflow when downgrading.

4. typeof(lua_Number): easy in C, but only when the host and the target follow the same FP standard; precision loss when upgrading (rare case); warn about non-integer numbers when downgrading to int32.

From all the discussions that I've seen about this issue on the mailing list, I see two likely viable approaches, assuming that you are unwilling to consider just shipping the uncompiled Lua scripts.

The first would be to fix the byte order as the compiled scripts are loaded. That turns out to be easier to do than you'd expect, as it can be done by replacing the low-level function that reads the script file without recompiling the core itself. In fact, it can even be done in pure Lua, by supplying your own chunk reader function to lua_load(). This should work as long as the only compatibility issue over your platforms is byte order.

The second is to patch the core itself to use a common representation for compiled scripts on all platforms. This has been described as possible by Luiz Henrique de Figueiredo:

.... I'm convinced that the best route to byte order or cross-compiling is third-party dump/undump pairs. The files ldump.c and lundump.c are completely replaceable; they export a single, well-defined, entry point. The format of precompiled chunks is not sacred at all; you can use any format, as long as ldump.c and lundump.c agree about it. (For instance, Rici Lake is considering writing a text format for precompiled chunks.) ....

Personally, I'd recommend giving serious consideration to not pre-compiling the scripts and thus avoid the platform portability issues entirely.

Edit: I've updated my description of the bytecode header thanks to lhf's comment. I hadn't read this part of the Lua source yet, and I probably should have checked it before being quite so assertive about what information is or is not present in the header.

Here is the fragment from lundump.c that forms a copy of the header matching the running platform for comparison to the bytecode being loaded. It is simply compared with memcmp() for an exact match to the header from the file, so any mismatch will cause the stock loader (luaU_undump()) to reject the file.

/*
* make header
*/
void luaU_header (char* h)
{
 int x=1;
 memcpy(h,LUA_SIGNATURE,sizeof(LUA_SIGNATURE)-1);
 h+=sizeof(LUA_SIGNATURE)-1;
 *h++=(char)LUAC_VERSION;
 *h++=(char)LUAC_FORMAT;
 *h++=(char)*(char*)&x;                         /* endianness */
 *h++=(char)sizeof(int);
 *h++=(char)sizeof(size_t);
 *h++=(char)sizeof(Instruction);
 *h++=(char)sizeof(lua_Number);
 *h++=(char)(((lua_Number)0.5)==0);             /* is lua_Number integral? */
}

As can be seen, the header is 12 bytes long and contains a signature (4 bytes, "<esc>Lua"), version and format codes, a flag byte for endianness, sizes of the types int, size_t, Instruction, and lua_Number, and a flag indicating whether lua_Number is an integral type.

This allows most platform distinctions to be caught, but doesn't attempt to catch every way in which platforms can differ.

I still stand by the recommendations made above: first, ship compilable sources; or second, customize ldump.c and lundump.c to store and load a common format, with the additional note that any custom format should redefine the LUAC_FORMAT byte of the header so as to not be confused with the stock bytecode format.

Answer 2

You may want to use a patched bytecode loader that supports different endianness. See this.