Networkx as a task queue?

Question

I have a directed acyclic graph in networkx. Each node represents a task and a nodes' predecessors are task dependencies (a given task cannot execute until its' dependencies have executed).

I'd like to 'execute' the graph in an asynchronous task queue, similar to what celery offers (so that I can poll jobs for their status, retrieve results etc). Celery doesnt offer the ability to create DAG's (as far as I know) and having the ability to move on to a task as soon as all dependencies are complete would be crucial (a DAG may have multiple paths and even if one task is slow/blocking, it may be possible to move on to other tasks etc).

Are there any simple examples as to how I could achieve this, or perhaps even integrate networkx with celery?

Answer 1

One library that you could use for this is taskgraph. It allows you to define a graph of tasks, and then executes these in a multi-thread/multi-process way. It avoids re-running tasks whose results are already up-to-date, similar to the make program.

To execute your networkx graph, you would iterate all the nodes in topological order, collect the imemdiate dependencies for each, and call task_graph.add_task. This function will return a handle to the newly added task, which lets you use it as dependency for subsequently added tasks (this is why the node iteration order is important)

For alternative solutions, see also this question.

Answer 2

I'm a bit late to the party, but one possibility is to use dask for constructing custom DAGs and then executing them, see https://docs.dask.org/en/stable/graphs.html.