R readBin vs. Python struct

Question

I am attempting to read a binary file using Python. Someone else has read in the data with R using the following code:

x <- readBin(webpage, numeric(), n=6e8, size = 4, endian = "little")
      myPoints <- data.frame("tmax" = x[1:(length(x)/4)],
                             "nmax" = x[(length(x)/4 + 1):(2*(length(x)/4))],
                             "tmin" = x[(2*length(x)/4 + 1):(3*(length(x)/4))],
                             "nmin" = x[(3*length(x)/4 + 1):(length(x))])

With Python, I am trying the following code:

import struct

with open('file','rb') as f:
    val = f.read(16)
    while val != '':
        print(struct.unpack('4f', val))
        val = f.read(16) 

I am coming to slightly different results. For example, the first row in R returns 4 columns as -999.9, 0, -999.0, 0. Python returns -999.0 for all four columns (images below).

Python output:

R output:

I know that they are slicing by the length of the file with some of the [] code, but I do not know how exactly to do this in Python, nor do I understand quite why they do this. Basically, I want to recreate what R is doing in Python.

I can provide more of either code base if needed. I did not want to overwhelm with code that was not necessary.

Answer 1

Deducing from the R code, the binary file first contains a certain number tmax's, then the same number of nmax's, then tmin's and nmin's. What the code does is reading the entire file, which is then chopped up in the 4 parts (tmax's, nmax's, etc..) using slicing.

To do the same in python:

import struct

# Read entire file into memory first. This is done so we can count
# number of bytes before parsing the bytes. It is not a very memory
# efficient way, but it's the easiest. The R-code as posted wastes even
# more memory: it always takes 6e8 * 4 bytes (~ 2.2Gb) of memory no
# matter how small the file may be.
#
data = open('data.bin','rb').read()

# Calculate number of points in the file.  This is 
# file-size / 16, because there are 4 numeric()'s per
# point, and they are 4 bytes each.
#
num = int(len(data) / 16)

# Now we know how much there are, we take all tmax numbers first, then
# all nmax's, tmin's and lastly all nmin's.

# First generate a format string because it depends on the number points
# there are in the file. It will look like: "fffff"
#
format_string = 'f' * num

# Then, for cleaner code, calculate chunk size of the bytes we need to 
# slice off each time.
#
n = num * 4     # 4-byte floats

# Note that python has different interpretation of slicing indices
# than R, so no "+1" is needed here as it is in the R code.
#
tmax = struct.unpack(format_string, data[:n])
nmax = struct.unpack(format_string, data[n:2*n])
tmin = struct.unpack(format_string, data[2*n:3*n])
nmin = struct.unpack(format_string, data[3*n:])

print("tmax", tmax)
print("nmax", nmax)
print("tmin", tmin)
print("nmin", nmin)

If the goal is to have this data structured as a list of points(?) like (tmax,nmax,tmin,nmin), then append this to the code:

print()
print("Points:")

# Combine ("zip") all 4 lists into a list of (tmax,nmax,tmin,nmin) points.
# Python has a function to do this at once: zip()
#
i = 0
for point in zip(tmax, nmax, tmin, nmin):
    print(i, ":", point)
    i += 1