No module named 'airflow' when initializing Apache airflow docker

Question

I am trying to run apache airflow as a docker on a Centos 7 machine. I followed all the instructions here:https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html when i am trying to initialize the docker by running docker-compose up airflow-init

i am getting this error

[root@centos7 centos]# docker-compose up airflow-init
Creating network "centos_default" with the default driver
Creating volume "centos_postgres-db-volume" with default driver
Creating centos_redis_1    ... done
Creating centos_postgres_1 ... done
Creating centos_airflow-init_1 ... done
Attaching to centos_airflow-init_1
airflow-init_1       | BACKEND=postgresql+psycopg2
airflow-init_1       | DB_HOST=postgres
airflow-init_1       | DB_PORT=5432
airflow-init_1       |
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-init_1       |     from airflow.__main__ import main
airflow-init_1       | ModuleNotFoundError: No module named 'airflow'
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-init_1       |     from airflow.__main__ import main
airflow-init_1       | ModuleNotFoundError: No module named 'airflow'
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-init_1       |     from airflow.__main__ import main
airflow-init_1       | ModuleNotFoundError: No module named 'airflow'
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-init_1       |     from airflow.__main__ import main
airflow-init_1       | ModuleNotFoundError: No module named 'airflow'
airflow-init_1       | Traceback (most recent call last):
airflow-init_1       |   File "/home/airflow/.local/bin/airflow", line 5, in <module>
airflow-init_1       |     from airflow.__main__ import main
airflow-init_1       | ModuleNotFoundError: No module named 'airflow'
centos_airflow-init_1 exited with code 1

i used the standard YAML file from here:https://airflow.apache.org/docs/apache-airflow/2.0.1/docker-compose.yaml i found that it's a known issue here:https://github.com/apache/airflow/issues/14520 but i could not understand how to solve this problem. any advice?

Answer 1

I solved this problem this way.

login with a non-root user.

find your user id :

echo $UID

create .env file and put these lines inside it . replace 4003 with your user id:

AIRFLOW_UID=4003
AIRFLOW_GID=0

If you have not created these directories, first create these and run docker-compose:

sudo mkdir ./dags ./logs ./plugins
sudo chmod 777 -R logs 
sudo docker-compose up airflow-init
sudo docker-compose up 

Answer 2

I found the problem. There is a bug on version 2.0.1 that doesn’t let you run the airflow containers using root. You have to run the installation under another user name (with sudo).