How can I run multiple jobs from a large queue at the same time in Perl?

Question

I would like to program job limits for the LSF command bsub into my Perl script which launches LSF jobs under the hood. If I have something like 2000 jobs, I would like to run at most 20 jobs at any given time. I have seen scripts that launch 20 jobs and then wait for them all to finish before launching another 20.

Answer 1

Several existing Perl modules, including Parallel::ForkManager and Forks::Super (of which I am the author) offer this functionality.

There is also an LSF::JobManager module that I don't know anything else about.

---

Parallel::ForkManager skeleton

use Parallel::ForkManager;
$pm = new Parallel::ForkManager(20);
foreach $job (@jobsToRun) {
    $pm->start and next;
    system("bsub -K $job");  # bsub -K job  to wait until job finishes, right?
    $pm->finish;
}

---

And in Forks::Super

use Forks::Super MAX_PROC => 20;
foreach $job (@jobsToRun) {
    fork { cmd => "bsub -K $job" };
}

---