Assigning tasks to specific machines with airflow

Question

I'm new to Airflow.

I have a DAG which contains a task that should run on a specific machine (EMR cluster in my case). How can I tell airflow where to run specific tasks so that every time it will run it will do so on that machine only?

Answer 1

Run your worker on that machine with a queue name. In the airflow cli you could do something like:

airflow worker -q my_queue

Then define that task to use that queue:

task = PythonOperator(
    task_id='task',
    python_callable=my_callable,
    queue='my_queue',
    dag=dag)