How to upgrade to Airflow 2.0?

Question

Now that Airflow 2.0 is released, we're excited to try out some the new features.

• What's the best way of upgrading from 1.10.11 to Airflow 2.0?
• Will my existing code work or will I be required to change my DAGs?

We'll start upgrading in our DEV environment for testing later this week.

Airflow 1.10.11 and local executor and Python3

Answer 1

The documentation lacks the information, how to exactly upgrade to 1.10.14 while the newer version is already available.

According to the PIP documentation (https://pip.pypa.io/en/stable/user_guide/#installing-packages) this should work:

python -m pip install apache-airflow==1.10.14

This seemed to work for me, but I was not able to start the websever after.

First, I had to upgrade the DB:

airflow upgradedb

Second, starting the webserver showed the problem that now the "secret_key" has to contain a real secret key.

Execute

openssl rand -hex 30

and add the hex key to the airflow.cfg file.

Then follow the remaining steps (including executing the check script) from the upgrade documentation.

As it is not described, either, the actual upgrade to 2.0 should work by using

pip install -U apache-airflow

Note especially the change in the DB upgrade command (airflow db upgrade instead of airflow upgradedb).

Regards, HerrB92

Answer 2

We have documented it at https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html

Step 1: Upgrade to Python 3

Step 2: Upgrade to Airflow 1.10.14 (a.k.a Airflow "bridge" release)

Step 3: Install and run the Airflow Upgrade check scripts (https://pypi.org/project/apache-airflow-upgrade-check/)

Step 4: Import Operators from Backport Providers

Step 5: Upgrade Airflow DAGs

Step 6: Upgrade Configuration settings

Step 7: Upgrade to Airflow 2.0

The upgrade-check package should help you in upgrading.

Read https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#step-3-install-and-run-the-upgrade-check-scripts