How to load several files in the same arrays in python with loadtxt?

Question

I have several table files and I load them with :

x, y, rho, phi = np.loadtxt(myfile, unpack=True)

Now, consider the case where I have myfile1 and myfile2 with the same format. How to load them in the same arrays (as if there was one file) ?

Answer 1

There are multiple ways to do this (e.g., you could load both arrays then concatenate them via x = numpy.concatenate((x1, x2)), etc.), but what I'd do is concatenate the files on the fly, and pass the result to loadtxt.

As the documentation says, fname can be:

File, filename, or generator to read. If the filename extension is .gz or .bz2, the file is first decompressed. Note that generators should return byte strings for Python 3k.

So, you just need a generator that generates myfile1, then myfile2.

The obvious way to do this is with itertools:

with open(myfile1, 'rb') as f1, open(myfile2, 'rb') as f2:
  x, y, rho, phi = numpy.loadtxt(itertools.chain(f1, f2))

You may notice that I left off the unpack=True. That's because unpack only works if you pass a filename, not a file object or generator. (It looks at the extension, not the file magic or anything fancy.)

If you know the files will always be gzip or bzip2 files, you can just substitute gzip.GzipFile or bz2.BZ2File for open above.

But if you need to deal with possibly compressed files, you have to do the same extension-checking that numpy does, then create the appropriate object (write a open_compressed function that wraps that up), at which point this becomes stubbornly sticking to the wrong solution. So, if that's an issue, I'd probably go with loading them separately and then doing numpy.concatenate.

Answer 2

You can use standard fileinput module:

import fileinput
import glob
import numpy as np
data = np.loadtxt(fileinput.input(glob.glob("*.dat")))