Accessing bitfields while reading/writing binary data structures

Question

I'm writing a parser for a binary format. This binary format involves different tables which are again in binary format containing varying field sizes usually (somewhere between 50 - 100 of them).

Most of these structures will have bitfields and will look something like these when represented in C:

struct myHeader
{
  unsigned char fieldA : 3
  unsigned char fieldB : 2;
  unsigned char fieldC : 3;
  unsigned short fieldD : 14;
  unsigned char fieldE : 4
}

I came across the struct module but realized that its lowest resolution was a byte and not a bit, otherwise the module pretty much was the right fit for this work.

I know bitfields are supported using ctypes, but I'm not sure how to interface ctypes structs containing bitfields here.

My other option is to manipulate the bits myself and feed it into bytes and use it with the struct module - but since I have close to 50-100 different types of such structures, writing the code for that becomes more error-prone. I'm also worried about efficiency since this tool might be used to parse large gigabytes of binary data.

Thanks.

Answer 1

Using bitstring (which you mention you're looking at) it should be easy enough to implement. First to create some data to decode:

>>> myheader = "3, 2, 3, 14, 4"
>>> a = bitstring.pack(myheader, 1, 0, 5, 1000, 2)
>>> a.bin
'00100101000011111010000010'
>>> a.tobytes()
'%\x0f\xa0\x80'

And then decoding it again is just

>>> a.readlist(myheader)
[1, 0, 5, 1000, 2]

Your main concern might well be the speed. The library is well optimised Python, but that's not nearly as fast as a C library would be.