Menu
I'm using Celery + Django + Supervisord and I'm trying to setup a "priority" by creating 3 different queues (as suggested at https://stackoverflow.com/a/15827160/54872).
Is there a way to start celery beat and workers for each queue in one command for supervisor? Or, do I need to make different supervisor conf files for each queue/worker pool and one for celery beat?
556
As for --concurrency celery by default uses multiprocessing to perform concurrent execution of tasks. The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of available CPU's if not set.
Integrate Celery into a Django app and create tasks. Write a custom Django Admin command. Schedule a custom Django Admin command to run periodically via Celery Beat.
When you run a celery worker, it creates one parent process to manage the running tasks. This process handles the book keeping features like sending/receiving queue messages, registering tasks, killing hung tasks, tracking status, etc.
Time Limits The time limit is set in two values, soft and hard . The soft time limit allows the task to catch an exception to clean up before it is killed: the hard timeout isn't catch-able and force terminates the task.
You can create program sections for each queue and combine them in a group section:
[program:worker1]
command=celery worker --queues=queue1
[program:worker2]
command=celery worker --queues=queue2
[group:workers]
programs=worker1,worker2
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
