Making np.loadtxt work with multiple possible delimiters

Question

I have a program that reads in data files and the user selects which column they want to use. I want it to be more universal with input files; sometimes, columns can look like this:

10:34:24.58  8.284  6.121

And sometimes they can look like this:

10 34 24.58  8.284  6.121

I want the program to recognize this as 5 columns in BOTH cases, instead of 5 columns for the first and 3 for the second. Basically, I want it to recognize white space as a delimiter and : as a delimiter as well.

Is there a simple way to do this? I know numpy takes a delimiter command, but as far as I'm aware it can only use one.

Answer 1

np.loadtxt (and genfromtxt) accept any iterable as input as long as it feeds it one line at a time. So the lines of your file can be passed through a function or generator that massages it in various ways. Here's a simple example

Define a pair of lines that simulates your file:

In [7]: txt="""10:34:24.58  8.284  6.121
   ...: 10 34 24.58  8.284  6.121
   ...: """

In [8]: txt=txt.splitlines()

In [9]: txt
Out[9]: ['10:34:24.58  8.284  6.121', '10 34 24.58  8.284  6.121']

If it weren't for the : I could pass this directly to loadtxt.

But let's pass the lines through a generator that replaces the ':' with a space. It could be a function (with yield). Here I'm using one of those new-fangled generator comprehension expressions:

In [10]: np.loadtxt((x.replace(b':',b' ') for x in txt))
Out[10]: 
array([[ 10.   ,  34.   ,  24.58 ,   8.284,   6.121],
       [ 10.   ,  34.   ,  24.58 ,   8.284,   6.121]])

with a file, this should work (iterating on an open file returns lines):

with open(filename) as f:
    A=np.loadtxt((x.replace(b':',b' ') for x in f))

regex would be useful for more elaborate replacements.