Any success story installing private dependency on GCP Composer Airflow?

Question

Background info

Normally within a container environment I can easily install my private dependency with a requirements.txt like this:

--index-url https://user:pass@some_repo.jfrog.io/some_repo/api/pypi/pypi/simple

some-private-lib

The package "some-private-lib" is the one I wanted to install.

Issue

Within the GCP Composer environment, I tried with the GCloud command ( gcloud composer environments update ENV_NAME --update-pypi-packages-from-file ./requirements.txt --location LOCATION), but it complained about requirements.txt not following formats defined in PEP-508. Then I found this OFFICIAL GUIDE on how to install dependency from private repo, but it isn't super clear. Following the instructions in the guide, I created a file pip.conf with following contents:

[global]
extra-index-url=https://user:pass@some_repo.jfrog.io/some_repo/api/pypi/pypi/simple

and then put it into my environment's GCS bucket: gs://us-central1-xxxx-bucket/config/pip/pip.conf.

Now I run the command (gcloud composer environments update ENV_NAME --update-pypi-packages-from-file ./requirements.txt --location LOCATION) again, with requirements.txt containing only one line: some-private-lib. It failed with a very opaque error: failed: Failed to install PyPI packages.

Question

What did I do wrong? Any other workarounds available? Thx!

Answer 1

We have also faced many issues when we try to install the packages in the Composer, then we containerized our code and used KubernetesPodOperator instead of Python operator.