Airflow connection to MySQL

Question

I would like do some ad hoc queries with my mysql database currently sitting on aws rds. I created a connection with all of the necessary credentials on the Airflow UI however the database did not show up under the Data Profiling>Ad hoc Query section.

Any help is appreciated. Thanks!

Answer 1

For the original question, the OP may simply need to install a python-mysql adapter.

I just ran into a similar issue.

For me, this issue was due to a lack of dependencies installed on my system.

As I was trying to connect to a Postgres database, I installed the python-postgres adapter, psycopg2:

pip install psycopg2

I restarted the Airflow web server, and Postgres connections began to populate within the Ad Hoc Query drop-down.

Here's how I identified this issue.

I was having this same problem trying to get a connection to an RDS Postgres server appear in the AdHoc query drop-down. After duplicating the existing sqlite_default connection, the drop-down appeared to include only Sqlite connections. This was with a near-vanilla default Airflow configuration. It seemed the connection was not listed because db.get_hook() returned None.

Stepping deeper into the code, I was able to identify that from airflow.hooks.postgres_hook import PostgresHook was failing with an error like:

*** ImportError: No module named 'psycopg2'

Using an interactive python debugger (eg. Pdb, via import pdb; pdb.set_trace()), the OP may find a similar error message, ie:

(Pdb) from airflow.hooks.mysql_hook import MySqlHook *** ImportError: No module named 'MySQLdb'