Menu
Within the Concurrency section of the Celery docs it states that:
...mix of both Eventlet and prefork workers, and route tasks according to compatibility or what works best
Source: http://celery.readthedocs.org/en/latest/userguide/concurrency/eventlet.html#concurrency-eventlet
This means that it's possible to have a worker use the gevent/eventlet pool implementation while another uses a prefork pool.
The pool implementation can be specified when creating multiple workers with celery multi:
celery -A proj multi start 2 -P gevent -c 1000
This starts 2 gevent workers, but how can I can specify pool implementation on a per worker basis when using celery multi, so that one worker uses a gevent pool and the other uses prefork?
The celery multi docs don't mention anything regarding this specific matter, and the source code (celery.bin.multi) didn't really show that this was possible (unless I misread/misunderstood the code).
287
So Celery (and other queue frameworks) has other benefits as well - Think of it as a 'task/function manager' rather then just a way of multithreading.
As for --concurrency celery by default uses multiprocessing to perform concurrent execution of tasks. The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of available CPU's if not set.
Introduction. The Eventlet homepage describes it as a concurrent networking library for Python that allows you to change how you run your code, not how you write it. It uses epoll(4) or libevent for highly scalable non-blocking I/O.
So currently the only way to specify per worker Pool implementations is by running independent celery worker commands:
$ celery -A proj worker start -P gevent -Q:queue1 -c 500
$ celery -A proj worker start -P prefork -Q:queue2 -c 4
celery multi does not support -P:worker1 gevent, -P:worker2 prefork. This makes things difficult when using the init.d script provided within the Celery docs to daemonize celeryd. So you either have to modify the init.d script or use something like Supervisor.
195
There is a way to start different implementations using celery multi. Just like Amir R. said about -Q option.
celery multi start 2 -A default_prefork_queue -Q:2 gevent_queue_name -P:2 gevent -c:2 1000
By default it's prefork option selected.
In example above celery creates one worker with gevent pool with 1000 limit and one prefork worker with default limit of processes.
More examples here
34
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
