Bash Script to Conda Install requirements.txt with PIP follow-up

Question

When installing a requirements.txt file for a Django app on a linux server I can run:

conda install --yes --file requirements.txt

This will crash if any of the packages are not available through Conda (PackageNotFoundError). This bash one liner is a cool way to go through the requirements.txt file one line at a time source:

while read requirement; do conda install --yes $requirement; done < requirements.txt

This installs all of the packages that are available through Conda without crashing on the first missing package. However, what I would like to is track the packages that fail by capturing the output from Conda and if there is a PackageNotFoundError then run pip install on the package.

I'm not great with bash, so hoping someone can suggest a hack. Another solution could be to just write out a new text file called pip-requirements.txt with the requirements that fail.

Answer 1

Personally I find the Anaconda environment & package management to be outstanding. So if you're using the conda command to update your packages in your python environment then I recommend using the environment.yml file instead of requirements.txt.

The environment.yml should looks like the following:

name: root            # default is root
channels:
- defaults
dependencies:         # everything under this, installed by conda
- numpy==1.13.3
- scipy==1.0.0
- pip:                # everything under this, installed by pip
  - Flask==0.12.2
  - gunicorn==19.7.1

The command to install:

conda env update --file environment.yml

Note:

Here we set name: root which is the default anaconda environment name. This is not standard practice of how to use the conda and environment.yml file. Ideally every python project should have its own environment.yml file with a project specific environment name i.e. name: project-name. Please go through https://conda.io/docs/user-guide/tasks/manage-environments.html on using Anaconda for package management.

Answer 2

Found a solution:

Run this to install with conda or pip if the package is not available to conda:

while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt 

Once it's done you can use conda to export the environment yaml:

conda env export > environment.yml

Answer 3

Redirect stderr to a file:

while read requirement; do conda install --yes $requirement; done < requirements.txt 2>error.log