Splitting command line args with GNU parallel

Question

Using GNU parallel: http://www.gnu.org/software/parallel/

I have a program that takes two arguments, e.g.

$ ./prog file1 file2 $ ./prog file2 file3 ... $ ./prog file23456 file23457 

I'm using a script that generates the file name pairs, however this poses a problem because the result of the script is a single string - not a pair. like:

$ ./prog "file1 file2" 

GNU parallel seems to have a slew of tricks up its sleeves, I wonder if there's one for splitting text around separators:

$ generate_file_pairs | parallel ./prog ?     # where ? is text under consideration, like "file1 file2" 

The easy work around is to split the args manually in prog, but I'd like to know if it's possible in GNU parallel.

Answer 1

You are probably looking for --colsep.

generate_file_pairs | parallel --colsep ' ' ./prog {1} {2}   

Read man parallel for more. And watch the intro video if you have not already done so http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 2

Quite late to the party here, but I bump into this problem fairly often and found a nice easy solution

Before passing the arg list to parallel, just replace all the spaces with newlines. I've found tr to be the fastest for this kind of stuff

Not working

echo "1 2 3 4 5"  | parallel echo -- -- 1 2 3 4 5 

Working

echo "1 2 3 4 5" | tr ' ' '\n' | parallel echo -- -- 1 -- 2 -- 3 -- 4 -- 5 

---

Protip: before actually running the parallel command, I do 2 things to check that the arguments have been split correctly.

1. Prepend echo in front of your bash command. This means that any commands that will eventually be executed will be printed for you to check first
2. Add a marker in the echo, this checks that the parallel split is actually working

---

> Note, this works best with small/medium argument lists. If the argument list is very large, probably best to just use a for loop to echo each argument to parallel