Which pool class should i use prefork, eventlet or gevent in celery?

Question

I have 3 remote workers, each one is running with default pool (prefork) and single task.

A single task is taking 2 to 5 minutes for completion as it runs on many different tools and inserts database in ELK.

worker command: celery -A project worker -l info

Which pool class should I use to make processing faster?

is there any other method to improve performance?

Answer 1

funny that this question scrolled by.

We just switched from eventlet to gevent. Eventlet caused hanging broker connections which ultimately stalled the workers.

General tips:

• Use a higher concurreny if you're I/O bound, I would start with 25, check the cpu load and tweak from there, aim for 99,9% cpu usage for the process.
• you might want to use --without-gossip and --without-mingle if your workforce grows.
• don't use RabbitMQ as your result backend (redis ftw!), but RabbitMQ is our first choice when it comes to a broker (the amqp emulation on redis and the hacky async-redis solution of celery is smelly and caused a lot of grief in our past).

---

More advanced options to tune your celery workers:

• pin each worker process to one core to avoid the overhead of moving processes around (taskset is your friend)
• if one worker isn't always working, consider core-sharing with one or two other processes, use nice if one process has priority