Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to migrate airflow variables between DEV and PROD environments?

We are using airflow to schedule our data pipelines, as part of it we also have added few connections and variables in airflow admin.

Everything worked fine in DEV, now we want to setup PROD environment. How do we migrate these values into PROD environment.

like image 987
Somashekar Muniyappa Avatar asked Feb 06 '18 06:02

Somashekar Muniyappa


People also ask

How do you change Airflow environment variables?

Setting Configuration Options cfg in your $AIRFLOW_HOME directory ( ~/airflow by default). This file contains Airflow's configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format: $AIRFLOW__{SECTION}__{KEY} (note the double underscores).

Where are Airflow environment variables stored?

Airflow uses Fernet to encrypt variables stored in the metastore database. It guarantees that without the encryption password, content cannot be manipulated or read without the key.


1 Answers

You can list or export variables and connection through the command line: https://airflow.apache.org/cli.html

Relevant commands:

  • airflow variables -e variables.json
  • airflow connections --list
like image 166
Antoine Augusti Avatar answered Oct 21 '22 22:10

Antoine Augusti