Does a modified command invocation tool – which dynamically regulates a job pool according to load – already exist?

Question

Fellow Unix philosophers,

I programmed some tools in Perl that have a part which can run in parallel. I outfitted them with a -j (jobs) option like make and prove have because that's sensible. However, soon I became unhappy with that for two reasons.

1. I specify --jobs=2 because I have two CPU cores, but I should not need to tell the computer information that it can figure out by itself.
2. Rarely runs of the tool occupy more than 20% CPU (I/O load is also little), wasting time by not utilising CPU to a better extent.

I hacked some more to add a load measuring, spawning additional jobs while there's still »capacity« until a load threshold is reached, this is when the number of jobs stays more or less constant, but when during the course of a run other processes with higher priority are in demand of more CPU, over time less new jobs are spawned and accordingly the number of jobs reduces.

Since this responsibility was repeated code in the tools, I factored out the scheduling aspect into a stand-alone tool following the spirit of nice et al.. The parallel tools are quite dumb now, they only have signal handlers through which they are told to increase or decrease the jobs pool, whereas the intelligence of load measuring and figuring out when to control the pool resides in the scheduler.

Taste of the tentative interface (I also want to provide sensible defaults so options can be omitted):

run-parallel-and-schedule-job-pool \
    --cpu-load-threshold=90% \
    --disk-load-threshold='300 KiB/s' \
    --network-load-threshold='1.2 MiB/s' \
    --increase-pool='/bin/kill -USR1 %PID' \
    --decrease-pool='/bin/kill -USR2 %PID' \
    -- \
    parallel-something-master --MOAR-OPTIONS

Before I put effort into the last 90%, do tell me, am I duplicating someone else's work? The concept is quite obvious, so it seems it should have been done already, but I couldn't find this as a single responsibility stand-alone tool, only as deeply integrated part of larger many-purpose sysadmin suites.

Bonus question: I already know runN and parallel. They do parallel execution, but do not have the dynamic scheduling (niceload goes into that territory, but is quite primitive). If against my expectations the stand-alone tool does not exists yet, am I better off extending runN myself or filing a wish against parallel?

Answer 1

some of our users are quite happy with condor. It is a system for dynamically distributing jobs to other workstations and servers according to their free computing resources.