What's the best way of executing tasks in parallel in Ksh and Perl?

Question

I have this large C++ project that I need to build on a platform that does not have a parallel make (like make -j on Linux). The server has 6 CPU's and I want to do a parallel build manually.

I can generate a task list like this for about 300 object files. I use the Makefile for the dependency checks and incremental build:

make -f Makefile obj1.o

make -f Makefile obj2.o

make -f Makefile obj3.o ...

How would I execute these tasks in parallel with no more then 6 tasks running at a time using Ksh and Perl? (Java or Python are not available :-( )

Answer 1

In Perl the you should look at Parallel::ForkManager. You could do something like this:

my @make_obj = qw(
  obj1.o
  obj2.o
  obj3.o
  ...
);

my $fm = $pm = new Parallel::ForkManager(6);
foreach my $obj (@make_obj) {
  $fm->start and next;
  system("make -f Makefile $make_obj");
  $fm->finish();
}

Answer 2

An alternative to forking is to run each make in its own thread.

use threads;

my $max_threads = 5;

my @targets = qw(obj1.o obj2.o obj3.o ...);
while(@targets) {
    my $num_threads = threads->list(threads::running);
    if( $num_threads < $max_threads ) {
        my $target = shift @targets;
        threads->create(sub { return system "make $target" });
    }
}

I am unfortunately hand waving around two bits. First is making the loop wait quiescently until a thread finishes. I believe this is accomplished using threads::shared's cond_wait() and a semaphore variable.

The second is getting the return value from make, so you know if something failed and to halt the build. To do that you'd have to join() each thread to get the result of system(), the process' exit code.

Sorry for the hasty reply. Hopefully the community will fill in the rest.