Error while connecting postgres db from airflow

Question

Using:

sql_alchemy_conn = db+postgresql://username:[email protected]:5432/airflow gives error:

sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:db.postgresql

and when using:

sql_alchemy_conn = postgresql+psycopg2://username:xxx@localhost:5432/airflow I could run the airlfow webserver -p 8080 but while running the scheduler: airflow scheduler it throws error:

ModuleNotFoundError: No module named 'MySQLdb'

packages versions I am using:

psycopg2==2.7.3.1
sqlalchemy==1.1.15
sqlalchemy-redshift== 0.7.0
apache_airflow=1.8.2

Earlier sql_alchemy_conn = db+postgresql://username:[email protected]:5432/airflow did worked for me - 2 months ago. I don't know what is the problem now.

Answer 1

finally,the following settings worked - in airflow.cfg:

sql_alchemy_conn = postgresql+psycopg2://scot:tiger@localhost:5432/airflow
celery_result_backend = db+postgres://scot:tiger@localhost:5432/airflow

It is important to note that dialect+driver are different for sql_alchemy_conn and celery_result_backend settings in airflow.cfg file, although they are pointing to the same database.