Run a looped process in bash across multiple cores

Question

I have a shell script that contains the following loop.

i=0  
upperlimit=$verylargevariable  
do  
   complexstuff RunManager file $i  
   i= 'expr $i +1'  
done

This script runs on a quad core machine, and according to top, uses about 15% of each core while executing one iteration of the loop. I'd like to distribute it across the four cores so that each iteration of the loop does complexstuff four times, one on each core, so the resources will be used more efficiently. We're talking about computation that currently takes several hours so efficiency is more than just good practice here. (The output of each iteration is obviously independent of the previous one.)

PS: Host is a server running Cent-OS, if that helps.

Answer 1

With GNU Parallel you can do:

seq $verylargevariable | parallel -j150% complexstuff RunManager file

The 150% will run 1.5 process per core so if it currently uses 15% this should give you around 100% on all 4 cores.

To learn more watch the intro videos: http://www.youtube.com/watch?v=OpaiGYxkSuQ