Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

GNU parallel: execute one command parallel for all files in a folder

Tags:

I am trying to parallelize particle simulations with different parameters to save some time. Therefore I wanted to use GNUparallel to run a bash script for the different parameters. The script reads a file and then performs the simulation eg :

$bash script <<< input file

However:-

$cd ~/parameter_files ls | parallel bash script <<< {}

does not work at all. I am a total newbie to Linux and GNUparallel, so hopefully someone can help.

like image 973
Physicus Avatar asked Nov 16 '17 14:11

Physicus


People also ask

How do I run multiple Linux commands in parallel?

On Linux, there are three ways to run multiple commands in a terminal: The Semicolon (;) operator. The Logical OR (||) operator. The Logical AND (&&) operator.

What does GNU parallel do?

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables.

Does Xargs run in parallel?

xargs will run the first two commands in parallel, and then whenever one of them terminates, it will start another one, until the entire job is done. The same idea can be generalized to as many processors as you have handy. It also generalizes to other resources besides processors.


1 Answers

You're almost right, use double quotes around the command

ls | parallel "bash script <<< {}"

Otherwise the here string <<< would feed the input into the parallel command rather than each individual bash script commands


I find this use case to be quite unusual however, since it means that basically your script is reading the filename string. If you just want to pass the files as arguments to the script, you can use

ls | parallel bash script

or if you want to pass the content of the files using standard input

ls | parallel "bash script < {}"
like image 121
etopylight Avatar answered Sep 22 '22 12:09

etopylight