Append multiple numpy files to one big numpy file in python

Question

I am trying to put many numpy files to get one big numpy file, I tried to follow this link Python append multiple files in given order to one big file and this is what I did:

import matplotlib.pyplot as plt 
import numpy as np
import os, sys

#Read in list of files. You might want to look into os.listdir()

path= "/home/user/Desktop/ALLMyTraces.npy/test"
#Test folder contains all my numpy file traces
traces= os.listdir(path)

# Create new File
f = open("/home/user/Desktop/ALLMyTraces.npy", "w")

for j,trace in enumerate(traces):

    # Find the path of the file
    filepath = os.path.join(path, trace)

    # Load file
    dataArray= np.load(filepath)
    f.write(dataArray)

File is created, and to verify that I have the good contents, I used this code:

import numpy as np
dataArray= np.load(r'/home/user/Desktop/ALLMyTraces.npy')
print(dataArray)

This error is produced as a result:

 dataArray= np.load(r'/home/user/Desktop/ALLMyTraces.npy')
  File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 401, in load
    "Failed to interpret file %s as a pickle" % repr(file))
IOError: Failed to interpret file '/home/user/Desktop/ALLMyTraces.npy' as a pickle

I don't know really the problem. Any help would be appreciated.

Answer 1

You should use numpy.save or numpy.savez to create pickled .npy or .npz binary files. Only those file can be read by numpy.load(). Since you are creating a text file using f.write(dataArray), np.load() is failing with the above mentioned error

Here is a sample

fpath ="path to big file"
npyfilespath ='path to nympy files to be merged '   
os.chdir(npyfilespath)

with open(fpath, 'wb') as f_handle:
    for npfile in glob.glob("*.npy"):

        # Find the path of the file
        filepath = os.path.join(path, npfile)
        print filepath
        # Load file
        dataArray= np.load(filepath)
        print dataArray
        np.save(f_handle,dataArray)
dataArray= np.load(fpath)
print dataArray

Just found that there is something really interesting in numpy load. It wont load all append arrays at once :). Read this post for more info.

This means , if you want to read all appended arrays , you need to load them multiple times.

f = open(fpath, 'rb')
dataArray= np.load(f) #loads first array
print dataArray
dataArray= np.load(f)  #loads Second array
print dataArray
dataArray= np.load(f) #loads Third array
print dataArray