Celery: long dedicated monolithic task vs short multiple tasks

Question

In my solution I use distributed tasks to monitor hardware instances for a period of time (say, 10 minutes). I have to do some stuff when:

• I start this monitoring session
• I finish this monitoring session
• (Potentially) during the monitoring session

Is it safe to have a single task run for the whole session (10 minutes) and perform all these, or should I split these actions into their own tasks?

The advantages of a single task, as I see it, are that it would be easier to manage and enforce timing constraints. But:

Is it a good idea to run a large pool of (mostly) asleep workers? For example, if I know that at most I will have 200 sessions open, I have a pool of 500 workers to ensure there are always available "session" seats?

Answer 1

There is no one-size-fits-all answer to this

• Dividing a big task A into many small parts (A¹, A², A³, …) will increase potential concurrency.

So if you have 1 worker instance with 10 worker threads/processes, A can now run in parallel using the 10 threads instead of sequentially on one thread.

The number of parts is called the tasks granularity (fine or coarsely grained).

• If the task is too finely grained the overhead of messaging will drag performance down.

Each part must have enough computation/IO to offset the overhead of sending the task message to the broker, possibly writing it to disk if there are no workers to take it, the worker to receive the message, and so on (do note that messaging overhead can be tweaked, e.g. you can have a queue that is transient (not persisting messages to disk), and send tasks that are not so important there).

• A busy cluster may make all of this moot

Maximum parallelism may already have been achieved if you have a busy cluster (e.g. 3 worker instances with 10 threads/processes each, all running tasks).

Then you many not get much benefit by dividing the task, but tasks doing I/O have a greater chance of improvement than CPU-bound tasks (split by I/O operations).

• Long running tasks are fine

The worker is not allergic to long running tasks, be that 10 minutes or an hour.

But it's not ideal either because any long running task will block that slot from finishing any waiting tasks. To mitigate this people use routing, so that you have a dedicated queue, with dedicated workers for tasks that must run ASAP.

-