Cannot install apache-airflow==2.0.0 because these package versions have conflicting dependencies

Question

I have a dockerfile:

FROM python:3.9.6-alpine3.14

ARG AIRFLOW_USER_HOME=/usr/local/airflow
ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}

RUN pip install apache-airflow[postgres,ssh,s3]==2.0.0 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.9.txt

#Add aws config profiles from local to docker machine
ADD ./environtment_config/airflow_config/entrypoint.sh /entrypoint.sh
ADD ./environtment_config/airflow_config/airflow.cfg ${AIRFLOW_USER_HOME}/airflow.cfg

RUN chown -R 777 ${AIRFLOW_USER_HOME}
RUN chmod -R 777 /entrypoint.sh

EXPOSE 8080 8081 5432 5555 8793

WORKDIR "/"
ENTRYPOINT ["/entrypoint.sh"]
#arg to entrypoint
CMD ["webserver"]

The error I get is:

ERROR: Cannot install apache-airflow[postgres,s3,ssh]==2.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[postgres,s3,ssh] 2.0.0 depends on pandas<2.0 and >=0.17.1
    The user requested (constraint) pandas==1.2.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

I can't understand what's the issue becuase the pandas version in the --constraint file is pandas==1.2.2. So it fullfill the needed dependency for apache-airflow.

To discard an issue related to the extra packages I also tested using:

RUN pip install apache-airflow==2.0.0 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.9.txt

And I get the same error.

Answer 1

See the installation documentation for Airflow 2.0.0: https://airflow.apache.org/docs/apache-airflow/2.0.0/installation.html - you need to use PIP 20.2.4 to install Airflow 2.0.0:

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 pip upgrade --pip==20.2.4 or, in case you use Pip 20.3, you need to add option --use-deprecated legacy-resolver to your pip install command.

Compatibility with newer PIP versions was fixed in 2.1.* series.

Just a question. Why would you want to install 2.0.0 version? Is there a particular reason? You are missing a lot of bug-fixes including some critical security fixes - for example this critical security fix applied in 2.1.2:

https://lists.apache.org/thread.html/r53d6bd7b0a66f92ddaf1313282f10fec802e71246606dd30c16536df%40%3Cusers.airflow.apache.org%3E

Airflow implements SemVer - which means that all 2.* releases are backwards-compatible. Even in 2.0.* series, there were 2.0.1 and 2.0.2 bugifxes, and we are just about to release 2.1.3. which provides another round of fixes to 2.1. I cannot imagine any reason why would you want to install Airflow 2.0.0. Could you explain why you need that?