Best way of creating large bit arrays in Lua

Question

I want to read a large binary file (1MB in size) into memory using Lua. The target device is mobile so I very much want to minimise the memory footprint.

From a quick look online it seems that Lua tables will use 16B for each sequential integer index (key) plus the space to store the value, which as I am storing binary data will hopefully only use 2 bits but lets just say 1 byte.

For 1e6 records that will be 1e6*17 =~ 17MB - which is huge!

From my brief reading it seems that I can use userdata to implement anything I want in C. I have not used C before but it seems that it would use

1b * 1e6 = 125kB

Shall i do this or have I got something very wrong / is there an easier way to do this.

Any advice or even name-calling for crappy calculations very much welcome :)

EDIT: Some interesting answers below about storing the data in a String (thanks!) and using bitwise ops. I just came accross an example in the PIL book (3rd edition pg293) that compares storing arrays of booleans in C so they use 3% of the mem. While this is cool and useful it may be overkill for me as the solutions below suggest I can fit in 1MB which is fine for me.

EDIT: Came across this C blob impl

EDIT: Solution - I read the file contents into a String as suggested and as Im using 5.1 had to use a 3rd party bit op lib - I went with a pure Lua implementation LuaBit. Thanks everyone!!

Answer 1

You can store a big blob in Lua string, it will work with any binary data. Now the question is what you want to do with the data. In any way, you can use string.byte to extract any individual byte, and use Lua's bit32 library to get down to bits. (For Lua 5.1 and older, you'll either have to write your own C routines, or use third party package.)

Answer 2

You can store the data in a string and manipulate it with the string library and Lua BitOp

Lua5.2's built-in bit32 library is preferred if available.