reading struct in python from created struct in c

Question

I am very new at using Python and very rusty with C, so I apologize in advance for how dumb and/or lost I sound.

I have function in C that creates a .dat file containing data. I am opening the file using Python to read the file. One of the things I need to read are a struct that was created in the C function and printed in binary. In my Python code I am at the appropriate line of the file to read in the struct. I have tried both unpacking the stuct item by item and as a whole without success. Most of the items in the struct were declared 'real' in the C code. I am working on this code with someone else and the main source code is his and has declared the variables as 'real'. I need to put this in a loop because I want to read all of the files in the directory that end in '.dat'. To start the loop I have:

for files in os.listdir(path):
  if files.endswith(".dat"):
    part = open(path + files, "rb")
    for line in part:

Which then I read all of the lines previous to the one containing the struct. Then I get to that line and have:

      part_struct = part.readline()
      r = struct.unpack('<d8', part_struct[0])

I'm trying to just read the first thing stored in the struct. I saw an example of this somewhere on here. And when I try this I'm getting an error that reads:

struct.error: repeat count given without format specifier

I will take any and all tips someone can give me. I have been stuck on this for a few days and have tried many different things. To be honest, I think I don't understand the struct module but I've read as much as I could on it.

Thanks!

Answer 1

Some C code:

#include <stdio.h>
typedef struct { double v; int t; char c;} save_type;
int main() {
    save_type s = { 12.1f, 17, 's'};
    FILE *f = fopen("output", "w");
    fwrite(&s, sizeof(save_type), 1, f);
    fwrite(&s, sizeof(save_type), 1, f);
    fwrite(&s, sizeof(save_type), 1, f);
    fclose(f);
    return 0;
}

Some Python code:

import struct
with open('output', 'rb') as f:
    chunk = f.read(16)
    while chunk != "":
        print len(chunk)
        print struct.unpack('dicccc', chunk)
        chunk = f.read(16)

Output:

(12.100000381469727, 17, 's', '\x00', '\x00', '\x00')
(12.100000381469727, 17, 's', '\x00', '\x00', '\x00')
(12.100000381469727, 17, 's', '\x00', '\x00', '\x00')

but there is also the padding issue. The padded size of save_type is 16, so we read 3 more characters and ignore them.

Answer 2

You could use ctypes.Structure or struct.Struct to specify format of the file. To read structures from the file produced by C code in @perreal's answer:

"""
struct { double v; int t; char c;};
"""
from ctypes import *

class YourStruct(Structure):
    _fields_ = [('v', c_double),
                ('t', c_int),
                ('c', c_char)]

with open('c_structs.bin', 'rb') as file:
    result = []
    x = YourStruct()
    while file.readinto(x) == sizeof(x):
        result.append((x.v, x.t, x.c))

print(result)
# -> [(12.100000381469727, 17, 's'), (12.100000381469727, 17, 's'), ...]

See io.BufferedIOBase.readinto(). It is supported in Python 3 but it is undocumented in Python 2.7 for a default file object.

struct.Struct requires to specify padding bytes (x) explicitly:

"""
struct { double v; int t; char c;};
"""
from struct import Struct

x = Struct('dicxxx')
with open('c_structs.bin', 'rb') as file:
    result = []
    while True:
        buf = file.read(x.size)
        if len(buf) != x.size:
            break
        result.append(x.unpack_from(buf))

print(result)

It produces the same output.

To avoid unnecessary copying Array.from_buffer(mmap_file) could be used to get an array of structs from a file:

import mmap # Unix, Windows
from contextlib import closing

with open('c_structs.bin', 'rb') as file:
    with closing(mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_COPY)) as mm: 
        result = (YourStruct * 3).from_buffer(mm) # without copying
        print("\n".join(map("{0.v} {0.t} {0.c}".format, result)))