Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Does the number of celeryd processes depend on the --concurrency setting?

We are running Celery behind Supervisor and start it with

celeryd --events --loglevel=INFO --concurrency=2

This, however, creates a process graph that is up to three layers deep and contains up to 7 celeryd processes (Supervisor spawns one celeryd, which spawns several others, which again spawn processes). Our machine has two CPU cores.

Are all of these processes working on tasks? Are maybe some of them just worker pools? How is the --concurrency setting connected to the number of processes actually spawned?

like image 650
Hannes Struß Avatar asked Aug 01 '12 09:08

Hannes Struß


People also ask

Is celery concurrent?

As for --concurrency celery by default uses multiprocessing to perform concurrent execution of tasks. The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of available CPU's if not set.

What is default concurrency in celery?

Concurrency settings The default is 4 (four messages for each process). The default setting seems pretty good here. However, if you have very long running tasks waiting in the queue and you have to start the workers, note that the first worker to start will receive four times the number of messages initially.

Does celery support multiprocessing?

Celery itself is using billiard (a multiprocessing fork) to run your tasks in separate processes.

How do you make multiple workers with celery?

You probably just need to add the --concurrency or -c argument when starting the worker to spawn multiple (parallel) worker instances. Show activity on this post. You can look for Canvas primitives there you can see how to make groups for parallel execution. class celery.


1 Answers

You shouldn't have 7 processes if --concurrency is 2.

The actual processes started is:

  • The main consumer process

    Delegates work to the worker pool

  • The worker pool (this is the number that --concurrency decides)

So that is 3 processes with a concurrency of two.

In addition a very lightweight process used to clean up semaphores is started if force_execv is enabled (which it is by default i you're using some other transport than redis or rabbitmq).

NOTE that in some cases process listings also include threads. the worker may start several threads if using transports other than rabbitmq/redis, including one Mediator thread that is always started unless CELERY_DISABLE_RATE_LIMITS is enabled.

like image 178
asksol Avatar answered Oct 09 '22 09:10

asksol