Running a particular task on airflow master node

Question

I have a dag with a list of tasks that are run using the celery executor on different worker nodes. However I would like to run one of the tasks on the master node. Is that possible?

Answer 1

Yes it is possible. You are able to set specific tasks to listen to specific queues in Celery. The airflow documentation covers it quite nicely but the gist of it is:

1. set a queue attribute on the operator representing the task you want to run on a specific node to a value different from the celery -> default_queue value in airflow.cfg

2. Run the worker process on your master node by specifying the queue it needs to listen on airflow worker -q queue_name. If you want your worker to listen to multiple queues, you can use a comma delimited list airflow worker -q default_queue,queue_name