celery multiple workers but one queue

Question

i am new to celery and redis.

I started up my redis server by using redis-server.

Celery was run using this parameter

celery -A proj worker 

There are no other configurations. However, i realised that when i have a long running job in celery, it does not process another task that is in the queue until the long running task is completed. My understanding is that since i have 8 cores on my CPU, i should be able to process 8 tasks concurrently since the default parameter for -c is the number of cores?

Am i missing something here ?

Answer 1

Your problem is classical, everybode met this who had long-running tasks.

The root cause is that celery tries to optimize your execution flow reserving some tasks for each worker. But if one of these tasks is long-running the others get locked. It is known as 'prefetch count'. This is because by default celery set up for short tasks.

Another related setting is a 'late ack'. By default worker takes a task from the queue and immediately sends an 'acknowledge' signal, then broker removes this task from the queue. But this means that more messages will be prefetched for this worker. 'late ack' enabled tells worker to send acknowledge only after the task is completed.

This is just in two words. You may read more about prefetch and late ack.

As for the solution - just use these settings (celery 4.x):

task_acks_late = True
worker_prefetch_multiplier = 1

or for previous versions (2.x - 3.x):

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

Also, starting the worker with parameter -Ofair does the same.