Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Airflow settings to run Celery workers in different Docker container?

I am running Airflow in docker container. I have created a separate container to run Postgres server and Rabbitmq server, connected these containers using docker network - by following this nice article. Now my Airflow docker container is running and connected to other containers using docker network - the process went smooth so far. The problem is how to run airflow webserver, airflow scheduler and airflow worker in the same container. After some research I found: it is recommended to run one service in one container. Now I have two solutions

  1. Run multiple services in the same Airflow container - which I could not figure out a simple way implement being a new bee in Docker.
  2. Create separate containers to run Celery worker and Airflow scheduler - but in the airflow.cfg file the setting related to Celery are: broker_url = 'amqp://guest:guest@ksaprice_rabbitmq:8080//', celery_result_backend = db+postgresql://developer:user889@ksaprice_postgres:5432/airflow. These settings refer to either database or rabbitmq which are already running different containers - they do not refer to ip/url which runs celery and scheduler and I assuming it is this way because celery and scheduler runs on the airflow server.

My questions are:

  1. Reffering to point 1: Is there a simple way to run airflow webserver, airflow scheduler and airflow worker commands in the same Airflow container?
  2. Refferring to point 2: Is there a way in airflow.cfg to configure airflow scheduler and airflow worker to run in separate docker containers - and link them using docker network?

I am new bee to Airflow and Docker.

like image 324
javed Avatar asked Sep 17 '25 08:09

javed


1 Answers

After spending lot of time I found the following answers:

  1. For the first question: To run multiple services on the same airflow_container do: docker exec -it airflow_container bash, now CLI will be attached to airflow_container then run airflow worker. Repeat the same process for airflow scheduler and airflow flower. Now you will have three different CLIs running three services on the same airflow_container - this is the simplest way I found .
  2. For the second question: There are options here:airflow cli like airflow webserver --hostname=some_host --port=some_port and airflow flower --hostname=some_host --port=some_port to run them on different severs. But for airflow worker there are no options to run on different server - may be there is some other way to run worker on different server.
like image 52
javed Avatar answered Sep 19 '25 05:09

javed