In celery, what would be the purpose of having multiple workers process the same queue?

Question

In the documentation for celeryd-multi, we find this example:

# Advanced example starting 10 workers in the background:
#   * Three of the workers processes the images and video queue
#   * Two of the workers processes the data queue with loglevel DEBUG
#   * the rest processes the default' queue.
$ celeryd-multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

( From here: http://docs.celeryproject.org/en/latest/reference/celery.bin.celeryd_multi.html#examples )

What would be a practical example of why it would be good to have more than one worker on a single host process the same queue, as in the above example? Isn't that what setting the concurrency is for?

More specifically, would there be any practical difference between the following two lines (A and B)?:

A:

$ celeryd-multi start 10 -c 2 -Q data

B:

$ celeryd-multi start 1 -c 20 -Q data

I am concerned that I am missing some valuable bit of knowledge about task queues by my not understanding this practical difference, and I would greatly appreciate if somebody could enlighten me.

Thanks!

Answer 1

What would be a practical example of why it would be good to have more than one worker on a single host process the same queue, as in the above example?

Answer：

So, you may want to run multiple worker instances on the same machine node if:

• You're using the multiprocessing pool and want to consume messages in parallel. Some report better performance using multiple worker instances instead of running a single instance with many pool workers.

• You're using the eventlet/gevent (and due to the infamous GIL, also the 'threads') pool), and you want to execute tasks on multiple CPU cores.

Reference： http://www.quora.com/Celery-distributed-task-queue/What-is-the-difference-between-workers-and-processes