Executing in background, but limit number of executions

Question

I have a program that performs some operations on a specified log file, flushing to the disk multiple times for each execution. I'm calling this program from a perl script, which lets you specify a directory, and I run this program on all files within the directory. This can take a long time because of all the flushes.

I'd like to execute the program and run it in the background, but I don't want the pipeline to be thousands of executions long. This is a snippet:

my $command = "program $data >> $log";
myExecute("$command");

myExecute basically runs the command using system(), along with some other logging/printing functions. What I want to do is:

my $command = "program $data & >> $log";

This will obviously create a large pipeline. Is there any way to limit how many background executions are present at a time (preferably using &)? (I'd like to try 2-4).

Answer 1

#!/bin/bash
#
# lets call this script "multi_script.sh"
#
#wait until there are less then 4 instances running
#polling with interval 5 seconds

while [ $( pgrep -c program ) -gt 4 ]; do sleep 5; done

/path/to/program "$1" &

Now call it like this:

my $command = "multi_script.sh $data" >> $log;

Your perl script will wait if the bash script waits.

positives: If a process crashes it will be replaced (the data goes, of course, unprocessed)

Drawbacks: It is important for your perl script to wait a moment between starting instances (maybe a sleep period of a second) because of the latency between invoking the script and passing the while loop test. If you spawn them too quickly (system spamming) you will end up with much more processes than you bargained for.