Bash: Logging stdout from multiple xargs parallel processes to separate log files

Question

I am processing a text file with multiple parallel processes spawned by xargs. I also need to capture the stdout from each process into a separate log file. Below is an example where the output from each process is interleaved into a single file -- not what I want.

Ideally, each logfile should be numbered by the file line number, that is, logfile-1, logfile-2, etc.

cat inputfile.txt | xargs -n 1 -P 8 ./myScript.sh | tee logfile

It would be nice to avoid an external wrapper script if possible, but if there is a way to wrap myScript with a here document, that would work.

Answer 1

Try this:

nl inputfile.txt | xargs -n 2 -P 8 sh -c './myScript.sh "$1" > logfile-$0'

This assumes each argument in inputfile.txt is on its own line and contains no spaces. The nl command numbers each line, which pairs each argument with a unique number. The xargs commands takes two arguments at time, the first the line number, the second the corresponding line from inputfile.txt, and passes them to sh. The sh command uses the arguments to generate the output file name and the argument to myScript.sh respectively.

Answer 2

You could use GNU Parallel instead and its -k option to keep the output in order, in a single log file:

cat input | parallel -k ./myScript.sh > file.log

You can add -j 8 after parallel to keep 8 cores busy, but it will keep all cores busy by default anyway.