Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to send periodic tasks to specific queue in Celery

By default Celery send all tasks to 'celery' queue, but you can change this behavior by adding extra parameter:

@task(queue='celery_periodic') def recalc_last_hour():     log.debug('sending new task')     recalc_hour.delay(datetime(2013, 1, 1, 2)) # for example 

Scheduler settings:

CELERYBEAT_SCHEDULE = {    'installer_recalc_hour': {         'task': 'stats.installer.tasks.recalc_last_hour',         'schedule': 15  # every 15 sec for test     }, } CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler" 

Run worker:

python manage.py celery worker -c 1 -Q celery_periodic -B -E 

This scheme doesn't work as expected: this workers sends periodic tasks to 'celery' queue, not 'celery_periodic'. How can I fix that?

P.S. celery==3.0.16

like image 485
Artem Mezhenin Avatar asked Jun 06 '13 12:06

Artem Mezhenin


People also ask

Can a celery task call another task?

To answer your opening questions: As of version 2.0, Celery provides an easy way to start tasks from other tasks. What you are calling "secondary tasks" are what it calls "subtasks".

What is beat in celery?

celery beat is a scheduler. It kicks off tasks at regular intervals, which are then executed by the worker nodes available in the cluster. By default the entries are taken from the CELERYBEAT_SCHEDULE setting, but custom stores can also be used, like storing the entries in an SQL database.


1 Answers

Periodic tasks are sent to queues by celery beat where you can do everything you do with the Celery API. Here is the list of configurations that comes with celery beat:

https://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#available-fields

In your case:

CELERYBEAT_SCHEDULE = {    'installer_recalc_hour': {         'task': 'stats.installer.tasks.recalc_last_hour',         'schedule': 15,  # every 15 sec for test         'options': {'queue' : 'celery_periodic'},  # options are mapped to apply_async options     }, } 
like image 138
abhi shukla Avatar answered Sep 22 '22 13:09

abhi shukla