Airflow and Docker Containers

Question

I am running airflow in containers AWS ECS, 1 scheduler, 2 web servers, and multiple celery workers.

From what I have seen the only thing that is affected when running them in containers is that the web servers are unable to access the workers' port on 8793 to retrieve logs from the workers.

Is that that only thing that is affected when running these in containers?

Answer 1

Yes because you can only map one port from docker container to host instance. I use a similar setup and logs are the only main issue I have faced. There are different ways to solve this though. You use other logging services on the container which push the logs to Cloudwatch or FLuentdb etc..