Parallel processing or threading in Shell scripting

Question

I am writing a script in shell in which a command is running and taking 2 min. everytime. Also, there is nothing we can do with this. But if i want to run this command 100 times in script then total time would be 200min. and this will create a big issue. Nobody want to wait for 200min. What i want is to run all 100 commands parallely so that output will come in 2min or may be some more time but dont take 200min.

it will be appreciated, if any body can help me on this in any way.

Answer 1

GNU Parallel is what you want, unless you want to reinvent the wheel. Here are some more detailed examples, but the short of it:

ls | parallel gzip # gzip all files in a directory

Answer 2

... run all 100 commands parallely so that output will come in 2min

This is only possible if you have 200 processors on your system.

There's no such utility/command in shell script to run commands in parallel. What you can do is run your command in background:

for ((i=0;i<200;i++))
do
   MyCommand &
done

With & (background), each execution is scheduled as soon as possible. But this doesn't guarantee that your code will be executed in less 200 min. It depends how many processors are there on your system.

If you have only one processor and each execution of the command (that takes 2min) is doing some computation for 2 min, then processor is doing some work, meaning there's no cycles wasted. In this case, running the commands in parallel is not going help because, there's only one processor which is also not free. So, the processes will be just waiting for their turn to be executed.

If you have more than one processors, then the above method (for loop) might help in reducing the total execution time.