Mismatch between sizeof in ctypes.struct (packed) and packed struct in C

Question

I have a packed structure in C which I would like to parse in Python. I have noted that there is a difference in size of structure with bitfields as returned by operator sizeof between C (using GCC 4.9.2) and ctypes library in Python 3.4.2.

The following C code prints 5 as expected:

#include <stdio.h>
#include <stdint.h>

typedef struct __attribute__((packed)) {
    uint32_t ch0 : 20;
    uint32_t ch1 : 20;
} pkt_t;

int main(){
    printf("sizeof(pkt_t): %d\n", sizeof(pkt_t));
    return 0;
}

While the (same) code in Python prints 8

import ctypes

class Packet(ctypes.LittleEndianStructure):
    _pack_ = 1
    _fields_ = [
        ('ch0', ctypes.c_uint32, 20),
        ('ch1', ctypes.c_uint32, 20),
    ]   

print(ctypes.sizeof(Packet()))

It looks like the _pack_ = 1 is equivalent to __attribute__((aligned(1))) in C, instead of __attribute__((packed, aligned(1))) which would make the struct to be packed as tightly as possible. Is there a way to enable packed attribute for ctypes Structure?

Answer 1

The ctypes docs https://docs.python.org/3/library/ctypes.html#structure-union-alignment-and-byte-order state clearly that

By default, Structure and Union fields are aligned in the same way the C compiler does it. It is possible to override this behavior by specifying a pack class attribute in the subclass definition. This must be set to a positive integer and specifies the maximum alignment for the fields. This is what #pragma pack(n) also does in MSVC.

This means they don't refer to what gcc does but to the MSVC #pragma pack, instead. See: https://docs.microsoft.com/en-us/cpp/preprocessor/pack?view=vs-2019

n

(Optional) Specifies the value, in bytes, to be used for packing. If the compiler option /Zp isn't set for the module, the default value for n is 8. Valid values are 1, 2, 4, 8, and 16. The alignment of a member is on a boundary that's either a multiple of n, or a multiple of the size of the member, whichever is smaller.

So, one reason for the observed behavior lies in the compiler specifics of MSVC vs. gcc.

The gcc docs https://gcc.gnu.org/onlinedocs/gcc-4.9.4/gcc/Type-Attributes.html#Type-Attributes (Sorry, docs for gcc-4.9.2 were not available on the gcc page), on the other hand cleary tells that:

This attribute(packed), attached to struct or union type definition, specifies that each member (other than zero-width bit-fields) of the structure or union is placed to minimize the memory required. When attached to an enum definition, it indicates that the smallest integral type should be used.

They do not tell nothing about boundary requirements, whereas the memory footprint is minimized, in this case. If it makes sense to place a bit field across a byte boundary may be topic of another discussion. It appears quite logical and consistent to me why __attribute__((aligned(1))) actually setups the struct as expected, because it explicitely enforces byte alignment of the struct members, which does not seem to be the case by specifying __attribute__((packed)). In the first case, the gcc compiler consequently enforces byte alignment for bit fields, too.

To summarize:

• MSVC #pragma pack(n) specifies the alignment of the struct members to be placed at memory offsets being multiples of n
• gcc's __attribute__ ((aligned (n))) asks for alignment of struct (members)
• gcc's __attribute__ ((__packed__))asks for minimizing memory footprint, even if that means that bitfields byte boundaries are crossed
• Combinations of gcc's packed and aligned attributes also asks for reduction of memory footprint but with the restriction of enforcing the alignment, too. So bitfield byte boundaries are not crossed