Celery command line parameter for number of worker processes when using gevent

Question

For multiprocessing pool, I could using the paramter '-c' to specify the number of worker processes. As the folloing shows.

celery worker -A celerytasks.celery_worker_init -c 5 --loglevel=info

But for gevent pool, the paramter '-c' is used to specify greenlet number. How can start multiple worker processes?

Or if I have to start mulitple wokers instead of one worker with multiple processes, could I do that with one command?

Answer 1

This command will run 1 worker with 5 pool processes. This means you have one worker able to execute 5 task at the time.

If you use gevent the only difference is that celery will use green threads instead of prefork to execute this tasks, but pay attention to the implication of this that you can read here (it's about eventlet but same concepts apply).

If you want to run multiple workers with different parameters (queue, concurrency) thant you need to use the celery multi command:

celery multi start 3 -c 3 -c:1 10
celery worker -n celery1@myhost -c 10
celery worker -n celery2@myhost -c 3
celery worker -n celery3@myhost -c 3