Gnu Parallel : nested parallelism

Question

Is it possible to call gnu parallel from within multiple runs of a script that are in-turn spawned by gnu parallel?

I have a python script that runs for 100s of sequential iterations, and somewhere within each iteration, 4 values are being computed in parallel (using gnu parallel). Now I want to spawn multiple such scripts at the same time, again, using gnu parallel. Is this possible? Will gnu parallel take care of good utilization of available cores?

For example, if in the inner loop, out of 4 values, 2 have been completed and 2 are running, so that a single script cannot proceed to the next iteration until all 4 values are computed. Will the two free cores be used for computing results for a different run of the script automatically? How can I specify the total number of cores available? In the inner call to parallel or outer call?

This questions shows it is possible to nest calls to parallel, but I'm not sure if this changes when I'm calling the nested parallel from inside a script.

PS: Thrashing is not a concern, I can use a LOT of cores from a large cluster.

PS2: gnu-parallel is an AWESOME tool... thanks! : )

Answer 1

Yes. GNU Parallel is designed (and tested heavily) to be able to be called from GNU Parallel - either directly or through a script.

If called directly you are likely to need to change -I. What does the second {} mean here (does it refer to the first or the second parallel?):

seq 10 | parallel 'seq {} | parallel echo {}'

Here it is very clear:

seq 10 | parallel 'seq {} | parallel -I // echo //'
seq 10 | parallel 'seq {} | parallel -I // echo {} //'

PS Good to hear you find GNU Parallel awesome. If you like GNU Parallel:

• Walk through the tutorial (http://www.gnu.org/software/parallel/parallel_tutorial.html)
• Give a demo at your local user group/team/colleagues
• Post the intro videos and tutorial on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
• Request or write a review for your favourite blog or magazine
• Invite me for your next conference

If you use GNU Parallel for research:

• Please cite GNU Parallel in you publications (use --bibtex)

If GNU Parallel saves you money:

• (Have your company) donate to FSF https://my.fsf.org/donate/