Python framework for task execution and dependencies handling

Question

I need a framework which will allow me to do the following:

• Allow to dynamically define tasks (I'll read an external configuration file and create the tasks/jobs; task=spawn an external command for instance)

• Provide a way of specifying dependencies on existing tasks (e.g. task A will be run after task B is finished)

• Be able to run tasks in parallel in multiple processes if the execution order allows it (i.e. no task interdependencies)

• Allow a task to depend on some external event (don't know exactly how to describe this, but some tasks finish and they will produce results after a while, like a background running job; I need to specify some of the tasks to depend on this background-job-completed event)

• Undo/Rollback support: if one tasks fail, try to undo everything that has been executed before (I don't expect this to be implemented in any framework, but I guess it's worth to ask..)

So, obviously, this looks more or less like a build system, but I don't seem to be able to find something that will allow me to dynamically create tasks, most things I've seem already have them defined in the "Makefile".

Any ideas?

Answer 1

Another option is to use make.

• Write a Makefile manually or let a python script write it
• use meaningful intermediate output file stages
• Run make, which should then call out the processes. The processes would be a python (build) script with parameters that tell it which files to work on and what task to do.
• parallel execution is supported with -j
• it also deletes output files if tasks fail

This circumvents some of the python parallelisation problems (GIL, serialisation). Obviously only straightforward on *nix platforms.