Python ctypes doesn't correctly return some values passed by reference

Question

I am trying to call a DLL from Python. The DLL is written in straight C and there is nothing terribly complicated about it. This is the kind of thing that ctypes is exactly made for. However I am having trouble getting correct values within a struct returned from the DLL function calls. The first value in the passed struct is returned correctly but others are not.

The code reads multiple messages from a buffer in memory. It uses a ReadFirst() / ReadNext() coding paradigm. There is a "current message" struct that is passed in to keep track of state.

Here is the C struct that holds the current ReadFirst() / ReadNext() state...

typedef struct
    {
    unsigned int            uMsgNum;
    uint32_t                ulCurrOffset;
    uint32_t                ulDataLen;
    Su1553F1_ChanSpec     * psuChanSpec;
    Su1553F1_Header       * psu1553Hdr;
    SuCmdWordU            * psuCmdWord1;
    SuCmdWordU            * psuCmdWord2;
    uint16_t              * puStatWord1;
    uint16_t              * puStatWord2;
    uint16_t                uWordCnt;
    uint16_t              * pauData;
    } Su1553F1_CurrMsg;

Here is the ctypes struct counterpart written in ctypes...

class CurrMsg_1553F1(ctypes.Structure):
    ''' Data structure for the current 1553 message info structure '''
    _pack_   = 1
    _fields_ = [("MsgNum",      ctypes.c_uint32),
                ("CurrOffset",  ctypes.c_uint32),
                ("DataLen",     ctypes.c_uint32),
                ("pChanSpec",   ctypes.POINTER(ChanSpec_1553F1)),
                ("p1553Hdr",    ctypes.c_void_p),
                ("pCmdWord1",   ctypes.POINTER(CmdWord)),
                ("pCmdWord2",   ctypes.POINTER(CmdWord)),
                ("pStatWord1",  ctypes.c_void_p),
                ("pStatWord2",  ctypes.c_void_p),
                ("WordCnt",     ctypes.c_uint16),
                ("pData",       ctypes.c_void_p)]

I'll provide more details below, but the crux of the problem is that calls to ReadFirst() / ReadNext() correctly return values for MsgNum but other values are garbage. Here is a sample output...

MsgNum 0  CurrOffset 40912728  DataLen 40912732  Messages = 3664897
MsgNum 1  CurrOffset 40912728  DataLen 40912748  Messages = 60417
MsgNum 2  CurrOffset 40912728  DataLen 40912748  Messages = 60417
etc.

MsgNum is correct and ReadFirst() / ReadNext() iterates the correct number of times. The values for CurrOffset, and DataLen however are garbage. (The Messages value is dereferenced from pointer to other memory. I don't think I have that right yet but I'll have to save that one until I get this other stuff fixed. If CurrOffset and DataLen are wrong then the pointer to pChanSpec is probably wrong also.)

Here are the interfaces for the functions...

EnI106Status enI106_Decode_First1553F1
    (SuI106Ch10Header * psuHeader,
     void             * pvBuff,
     Su1553F1_CurrMsg * psuMsg);

EnI106Status enI106_Decode_Next1553F1
    (Su1553F1_CurrMsg * psuMsg);

These DLL functions get called (along with debugging print statements) by...

def __init__(self, PacketIO):
    self.CurrMsg  = CurrMsg_1553F1()

def Decode_First1553F1(self):
    Status = self.PacketIO._IrigDataDll.enI106_Decode_First1553F1(ctypes.byref(self.PacketIO.Header), ctypes.byref(self.PacketIO.Buffer),  ctypes.byref(self.CurrMsg))
    print "MsgNum %d  CurrOffset %d  DataLen %d  Messages = %d" % \
        (Decode1553.CurrMsg.MsgNum, Decode1553.CurrMsg.CurrOffset, Decode1553.CurrMsg.DataLen, Decode1553.CurrMsg.pChanSpec.contents.MsgCnt)
    return Status

def Decode_Next1553F1(self):
    Status = self.PacketIO._IrigDataDll.enI106_Decode_Next1553F1(ctypes.byref(self.CurrMsg))
    print "MsgNum %d  CurrOffset %d  DataLen %d  Messages = %d" % \
        (Decode1553.CurrMsg.MsgNum, Decode1553.CurrMsg.CurrOffset, Decode1553.CurrMsg.DataLen, Decode1553.CurrMsg.pChanSpec.contents.MsgCnt)
    return Status

The C DLL is compiled under Visual Studio 2005. I've used this DLL a lot with various C and C++ .NET programs with no problems.

I have used ctypes and structs to return values in other code and it has seemed to work just fine. Since the first value in the struct is OK but subsequent values are way off this "feels" like a data alignment issue. I have set pack=1 in the ctypes struct and have set byte alignment in the compiled DLL code. I have done sizeof printouts of each struct and verified they are the expected size.

I've beat my head against the wall on this one for a while now and am out of ideas. Any thoughts on what to try next?

Answer 1

I note the C struct's first element is unsigned int, but your python Structure is c_uint32. You're sure they're the same size on your architecture?

If that isn't any help, I would do the following to trace the source of the problem:

First, I would remove _pack_ from the python declaration and any #pragma pack or other alignment manipulations from the C source. Start from a known place, native alignment.

Then...

I'd write a C program to iterate over the struct and print the offsets of each member with offsetof(struct, field). Also, this program should make a call to First() and a handful of calls to Next() and write the structs, as is, in binary, to a file.

Next, write a python script using the struct module and, iterating over the structs you wrote out from the C program, and armed with the knowledge of the field sizes and alignments, make calls to struct.unpack(...) finding the format string that properly defines the struct.

With that knowledge you should be able to properly code your ctypes.Structure