Skip a specified number of columns with numpy.genfromtxt()

Question

I have a large table (numbers in text format) that I would like to load with numpy.genfromtxt(). I would like to ignore the first n columns, say 5. I do not know the size of the table (number of row or columns) in advance.

I saw that genfromtxt() has an option skip_header that allows to skip a specified number of header rows, but it seems there is no such option for columns. There is a usecols option but there I must specify the column numbers I want to keep, rather than those I want to discard (I do not know this number in advance).

Obviously I could just load the whole thing and then throw away the first n columns, but this is not elegant and is wasteful in terms of memory.

Also I could peak into the file, find the number of columns, and then construct the usecols argument, but that is rather messy.

Any ideas on how to solve this elegantly? Is there some hidden/undocumented argument that I can use?

Answer 1

For older versions of numpy, peeking at the first line to discover the number of columns is not that hard:

import numpy as np
with open(fname, 'r') as f:
    num_cols = len(f.readline().split())
    f.seek(0)
    data = np.genfromtxt(f, usecols = range(5,num_cols))
print(data)

Answer 2

In newer versions of Numpy, np.genfromtxt can take an iterable argument, so you can wrap the file you're reading in a generator that generates lines, skipping the first N columns. If your numbers are space-separated, that's something like

np.genfromtxt(" ".join(ln.split()[N:]) for ln in f)