Run script in multiple machines in parallel

Question

I am interested to know the best way to start a script in the background in multiple machines as fast as possible. Currently, I'm doing this

Run for each IP address

ssh user@ip -t "perl ~/setup.pl >& ~/log &" &

But this takes time as it individually tries to SSH into each one by one to start the setup.pl in the background in that machine. This takes time as I've got a large number of machines to start this script on.

I tried using GNU parallel, but couldn't get it to work properly:

seq COUNT | parallel -j 1 -u -S ip1,ip2,... perl ~/setup.pl >& ~/log

But it doesn't seem to work, I see the script started by GNU parallel in the target machine, but it's stagnant. I don't see anything in the log.

What am I doing wrong in using the GNU parallel?

Answer 1

GNU Parallel assumes per default that it does not matter which machine it runs a job on - which is normally true for computations. In your case it matters greatly: You want one job on each of the machine. Also GNU Parallel will give a number as argument to setup.pl, and you clearly do not want that.

Luckily GNU Parallel does support what you want using --nonall:

http://www.gnu.org/software/parallel/man.html#example__running_the_same_command_on_remote_computers

I encourage you to read and understand the rest of the examples, too.

Answer 2

I recommend that you use pdsh

It allows you to run the same command on multiple machines

Usage:

pdsh -w machine1,machine2,...,machineN <command>

It might not be included in your distribution of linux so get it through yum or apt