I am running my server using this command:
celery worker -Q q1,q2 -c 2
which shows that my server will handle all the tasks on queues q1
and q2
, and I have 2 workers running. My server should support 2 different tasks:
@celery.task(name='test1')
def test1():
print "test1"
time.sleep(3)
@celery.task(name='test2')
def test2():
print "test2"
If I send my test1
tasks to queue q1
and test2
to q2
, both workers will run both tasks. So the result will be:
test1
test2
test1
test2
...
Now what I need is one of my workers handle test1
and the other one handles test2
. One solution is to run two celery workers like this:
celery worker -Q q1 -c 1
celery worker -Q q2 -c 1
And each one handles 1 queue. But I would like to have them cleaner and use -c 2
. I found Celery Routing but am not sure if that is what I want.
You can look for Canvas primitives there you can see how to make groups for parallel execution. class celery. group(task1[, task2[, task3[, … taskN]]]) Creates a group of tasks to be executed in parallel.
Celery provides a way to both design a workflow for coordination and also execute tasks in parallel.
You can convert any function into a Celery task using the @app. task decorator which adds all the necessary functionality required for our existing function to run as a task. You can copy and paste the final code into a new file named tasks.py to follow the instructions given in the next section.
Answer: Yes, but make sure it's unique, as the behavior for two tasks existing with the same id is undefined.
I found the answer and I am putting it here in case someone else wanted to do the same:
Instead of using celery worker -Q q1,q2 -c 2
, celery multi
could be used:
celery multi start 2 -Q:1 q1 -Q:2 q2 -c:1 1 -c:2 1
Which says that we have 2 queues: -Q:1 q1
means queue #1 with name of q1
and same for q2
and we have different concurrencies for each queue, -c:1 1
means the first celery worker has a concurrency of 1.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With