struct.error: unpack requires a string argument of length 4

Question

Python says I need 4 bytes for a format code of "BH":

struct.error: unpack requires a string argument of length 4

Here is the code, I am putting in 3 bytes as I think is needed:

major, minor = struct.unpack("BH", self.fp.read(3))

"B" Unsigned char (1 byte) + "H" Unsigned short (2 bytes) = 3 bytes (!?)

struct.calcsize("BH") says 4 bytes.

EDIT: The file is ~800 MB and this is in the first few bytes of the file so I'm fairly certain there's data left to be read.

Answer 1

The struct module mimics C structures. It takes more CPU cycles for a processor to read a 16-bit word on an odd address or a 32-bit dword on an address not divisible by 4, so structures add "pad bytes" to make structure members fall on natural boundaries. Consider:

struct {                   11
    char a;      012345678901
    short b;     ------------
    char c;      axbbcxxxdddd
    int d;
};

This structure will occupy 12 bytes of memory (x being pad bytes).

Python works similarly (see the struct documentation):

>>> import struct
>>> struct.pack('BHBL',1,2,3,4)
'\x01\x00\x02\x00\x03\x00\x00\x00\x04\x00\x00\x00'
>>> struct.calcsize('BHBL')
12

Compilers usually have a way of eliminating padding. In Python, any of =<>! will eliminate padding:

>>> struct.calcsize('=BHBL')
8
>>> struct.pack('=BHBL',1,2,3,4)
'\x01\x02\x00\x03\x04\x00\x00\x00'

Beware of letting struct handle padding. In C, these structures:

struct A {       struct B {
    short a;         int a;
    char b;          char b;
};               };

are typically 4 and 8 bytes, respectively. The padding occurs at the end of the structure in case the structures are used in an array. This keeps the 'a' members aligned on correct boundaries for structures later in the array. Python's struct module does not pad at the end:

>>> struct.pack('LB',1,2)
'\x01\x00\x00\x00\x02'
>>> struct.pack('LBLB',1,2,3,4)
'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04'

Answer 2

By default, on many platforms the short will be aligned to an offset at a multiple of 2, so there will be a padding byte added after the char.

To disable this, use: struct.unpack("=BH", data). This will use standard alignment, which doesn't add padding:

>>> struct.calcsize('=BH')
3

The = character will use native byte ordering. You can also use < or > instead of = to force little-endian or big-endian byte ordering, respectively.