Menu
I tried using ENV variable to configure connection urls, I have an ami that is preconfigured with alchemy_conn and broker_url etc, I have written environment variables to /etc/environment in the instances being spun up from the amis to override the properties in airflow.cfg file. I was able to access the variables from python code as well.
But it doesn't seem to take effect while running ariflow, tried restarting the process as well but didn't work, it still points to the one in airflow.cfg file
603
As per this answer, the variables should be put in /etc/default/airflow (on Debian/Ubuntu) or /etc/sysconfig/airflow (on Centos/Redhat). Show activity on this post. If you are just running a local instance you should be able to use environment variables like you expect.
The Apache Airflow configuration file is located at /opt/bitnami/airflow/airflow. cfg.
What is $AIRFLOW_HOME? $AIRFLOW_HOME is a location that contains all configuration files, DAGs, plugins, and task logs. It is an environment variable set to /usr/lib/airflow for all machine users.
parallelism is the max number of task instances that can run concurrently on airflow. This means that across all running DAGs, no more than 32 tasks will run at one time.
This is in contrast with the way airflow.cfg parameters are stored, where double underscores surround the config section name. Variables set using Environment Variables would not appear in the Airflow UI but you will be able to use them in your DAG file. Airflow uses Fernet to encrypt variables stored in the metastore database.
Setting Configuration Options¶ The first time you run Airflow, it will create a file called airflow.cfgin your $AIRFLOW_HOMEdirectory (~/airflowby default). This file contains Airflow's configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format:
Variables set using Environment Variables would not appear in the Airflow UI but you will be able to use them in your DAG file. Airflow uses Fernet to encrypt variables stored in the metastore database. It guarantees that without the encryption password, content cannot be manipulated or read without the key.
Apache Airflow configuration options can be attached to your Amazon Managed Workflows for Apache Airflow (MWAA) environment as environment variables. You can choose from the suggested dropdown list, or specify custom configuration options for your Apache Airflow version on the Amazon MWAA console.
The issue is probably that system services do not automatically pick up environment from /etc/environment that is just for interactive sessions.
If you are using systemd you can add EnvironmentFile=/etc/environment, though more "typical" would be to put the service specific config in /etc/default/airflow (on Debian/Ubuntu) or /etc/sysconfig/airflow (on Centos/Redhat)
The example systemd scripts from Airflow already have this for Centos: https://github.com/apache/incubator-airflow/blob/faa9a5266c0b2e68693dd106b5cb46d30770dadc/scripts/systemd/airflow-webserver.service#L20
118
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
