Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Assign different tasks to different celery workers

Tags:

python

celery

I am running my server using this command:

celery worker -Q q1,q2 -c 2

which shows that my server will handle all the tasks on queues q1 and q2, and I have 2 workers running. My server should support 2 different tasks:

@celery.task(name='test1')
def test1():
    print "test1"
    time.sleep(3)

@celery.task(name='test2')
def test2():
    print "test2"

If I send my test1 tasks to queue q1 and test2 to q2, both workers will run both tasks. So the result will be:

test1
test2
test1
test2
...

Now what I need is one of my workers handle test1 and the other one handles test2. One solution is to run two celery workers like this:

celery worker -Q q1 -c 1
celery worker -Q q2 -c 1

And each one handles 1 queue. But I would like to have them cleaner and use -c 2. I found Celery Routing but am not sure if that is what I want.

like image 762
AliBZ Avatar asked Oct 29 '13 22:10

AliBZ


People also ask

How do you run multiple celery workers?

You can look for Canvas primitives there you can see how to make groups for parallel execution. class celery. group(task1[, task2[, task3[, … taskN]]]) Creates a group of tasks to be executed in parallel.

Does celery run tasks in parallel?

Celery provides a way to both design a workflow for coordination and also execute tasks in parallel.

How do I add tasks to a celery queue?

You can convert any function into a Celery task using the @app. task decorator which adds all the necessary functionality required for our existing function to run as a task. You can copy and paste the final code into a new file named tasks.py to follow the instructions given in the next section.

Is celery task ID unique?

Answer: Yes, but make sure it's unique, as the behavior for two tasks existing with the same id is undefined.


1 Answers

I found the answer and I am putting it here in case someone else wanted to do the same:

Instead of using celery worker -Q q1,q2 -c 2, celery multi could be used:

celery multi start 2 -Q:1 q1 -Q:2 q2 -c:1 1 -c:2 1

Which says that we have 2 queues: -Q:1 q1 means queue #1 with name of q1 and same for q2 and we have different concurrencies for each queue, -c:1 1 means the first celery worker has a concurrency of 1.

like image 192
AliBZ Avatar answered Oct 03 '22 13:10

AliBZ