How to avoid running a specific task simultaneously in Luigi with multiple workers

Question

I use Luigi to build data analysis tasks including plotting by matplotlib.

It seems concurrent runs of matplotlib plotting causes a problem, which causes returning from the task prematurely, doing nothing, for some reason. (Looks like this is the problem with matplotlib, though I might be wrong.)

To solve this issue, I want to avoid running multiple workers for only that plotting task simultaneously, while running other tasks in multiple workers. How can I do that?

Answer 1

You can use resources for that. On /etc/luigi/client.cfg configure a resource like:

[resources]
mathplotlib: 1

And then, modify your task this way:

class MyTask(luigi.Task):
    resources = {"mathplotlib": 1}

If you have muliple machines running luigi workers and you want that only one worker across all machines may be using a given resource, then you can take a look a this solution.