Unpacking double values from binary file

Question

I understand that there are issues reading C double values in Python. In my program, I am reading from a binary file and convert numeric values to integers of various sizes without an problems. I use the code below to read the double values.

peakDescriptor["area"] = struct.unpack("d",file.read(8))

There is a massive discrepancy between what the values should be and what I get. The first table below is what I get:
methane 3.6368230562528605e-307
ethane -8.243249632731949e+306
propane 1.839329701286865e-60
2-methylpropane -2.55127317345224e-306
butane 3.737451552798833e+59
...

And this table shows, what the values should be:
methane 97.25
ethane 426.50
propane 2755.60
2-methylpropane 3390.25
butane 10906.60
...

How can I correctly read these numbers?

My code can be found here

The raw and result files are here and here

Please let me know if you have problems accessing the files!

P.S. I have tried changing the formatting string to include a ">" sign as per the struct documentation - this still results in unexpected values, as well as a number of NaN's!

Answer 1

Short answer: you need to interpret your bytes using the correct byte order. It turns out that in this case, the correct byte order is neither little-endian (order 01234567) nor big-endian (76543210), but the order 32107654, so you need a tiny bit of preprocessing before you can use the struct module. See the function interpret_float below.

In more detail: looking first at the methane value, and making the (fairly safe) guess that the machine you're using is little endian, the bytes that you read in from the file look like this:

>>> field = struct.pack('<d', 3.6368230562528605e-307)
>>> field
b'\x00\x00\x003@XP\x00'

As you've already discovered, trying to interpret these bytes directly as an IEEE 754 binary64 (i.e., double precision) floating-point value assuming either little-endian or big-endian byte order doesn't produce plausible values:

>>> struct.unpack('<d', field)[0]
3.6368230562528605e-307
>>> struct.unpack('>d', field)[0]
1.08755143765e-312

However, there's a suspicious similarity between the bytes of field and the bytes of the expected value of 97.25. It's slightly easier to see if you expand the bytes out to see their integer values:

>>> list(struct.pack('<d', 97.25))
[0, 0, 0, 0, 0, 80, 88, 64]
>>> list(field)
[0, 0, 0, 51, 64, 88, 80, 0]

Not a perfect match, but the 0, 80, 88, 64 sequence in the bytes for 97.25 looks suspiciously like the exact reversal of the sequence 64, 88, 80, 0 in the second. Besides little-endian and big-endian, there's another pair of byte orders for IEEE 754 double-precision floats that turn up occasionally (usually on ARM hardware), and that's word-swapped little-endian, or word-swapped big-endian (both of which are sometimes referred to as mixed-endian or middle-endian). In your case, it looks as though the bytes you have are in order 32107654, where 7 denotes the most significant byte (the one containing the sign bit and the most significant 7 bits of the biased exponent), and 0 the least significant byte (containing the 8 least significant bits of the fraction). So if we swap the two words, we should be able to interpret as regular big-endian:

>>> def interpret_float(x):
...     return struct.unpack('>d', x[4:] + x[:4])
... 
>>> interpret_float(field)
(97.25000000000072,)

That looks much more promising! Let's try the same on the next couple of values. You didn't give the original bytes for these, so again I need to reverse-engineer them from the bad values you gave.

>>> ethane_field = struct.pack('<d', -8.243249632731949e+306)
>>> interpret_float(ethane_field)
(426.4999999999999,)
>>> propane_field = struct.pack('<d', 1.839329701286865e-60)
>>> interpret_float(propane_field)
(2755.600000000001,)

From these, it looks as though our guess about a byte order of 32107654 was correct.

If my original guess was wrong and you are in fact on a big-endian machine, or you're on a little-endian machine and the values you show were obtained by doing a struct.unpack('>d', ...) rather than a plain old struct.unpack('d', ...), then the byte order is 45670123, and you'll need to replace the '>d' format in interpret_float with '<d' instead.

You can find out what byte order your host machine is using by looking at sys.byteorder in Python. On my machine, and on any other x86-64-based machine, it gives 'little':

>>> import sys
>>> sys.byteorder
'little'