Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

File I/O in gnu parallel

I have a program that takes a single argument. I am using gnu parallel to perform parameter sweeps on this argument. Each run generates a single result, and I want to append all results into a single file, say Results.txt.

What would be a correct way to do this?

I should not have each instance open the file and write to it, as this could create conflicts and also mess up the order of results. The only way I can think of doing this is having each run generate its output in a file with a unique name, and then , when gnu parallel finishes running, merge the results into a single file using a script.

  1. Is there a simpler way of achieving this?
  2. What happens when multiple instances write to/read from the same file? Does gnu parallel create multiple copies, one for each instances, as it does for stdout and stderror?

thanks

like image 809
Neha Karanjkar Avatar asked Dec 20 '22 21:12

Neha Karanjkar


1 Answers

If your command sends the result to stdout (standard output) the solution is trivial:

seq 1000 | parallel echo > Results.txt

GNU Parallel guarantees the output will not be mixed.

like image 171
Ole Tange Avatar answered Jan 28 '23 20:01

Ole Tange