Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Export Conda Environment with minimized requirements

The typical command to export a Anaconda environment to a YAML file is:

conda env export --name my_env > myenv.yml

However, one huge issue is the readbility of this file as it includes hard specifications for all of the libraries and all of their dependencies. Is there a way for Anaconda to export a list of the optimally smallest subset of commands that would subsume these dependencies to make the YAML more readable? For example, if all you installed in a conda environment was pip and scipy, is there a way for Anaconda to realize that the file should just read:

name: my_env
channels:
  - defaults
dependencies:
  - scipy=1.3.1
  - pip=19.2.3

That way, the anaconda environment will still have the exact same specification, if not an improved on (if an upstream bug is fixed) and anyone who looks at the yml file will understand what is "required" to run the code, in the sense that if they did want to/couldn't use the conda environment they would know what packages they needed to install?

like image 961
physincubus Avatar asked Oct 09 '20 17:10

physincubus


People also ask

Can you export a conda environment?

With conda, you can create, export, list, remove, and update environments that have different versions of Python and/or packages installed in them. Switching or moving between environments is called activating the environment. You can also share an environment file.

Can I use pip in a conda environment?

You can install pip in the current conda environment with the command conda install pip , as discussed in Using pip in an environment. If there are instances of pip installed both inside and outside the current conda environment, the instance of pip installed inside the current conda environment is used.

Do conda environments inherit from base?

If you created the new environment using something like conda create --name dell_proj , it will not inherit packages from the base environment. You would have to install the packages you want using conda install .

How do I export an environment from Conda?

Step 1: Activate the environment to export by typing: conda activate myenv Step 2: Export your active environment to a new file by typing: conda env export > environment.yml On Mac, do source env export > environment.yml Go the directory you are currently working on and you will find your environment.yml file. Step 3: .

How to associate environment variables with an environment in conda?

Conda env will export or create environments based on a file with conda and pip requirements. If you want to associate environment variables with an environment, you can use the config API.

How do I use Conda and Pip requirements in Python?

Package requirements can be passed to conda via the --file argument. Pip accepts a list of Python packages with -r or --requirements. Conda env will export or create environments based on a file with conda and pip requirements. If you want to associate environment variables with an environment, you can use the config API.

How do I use the SPEC file in conda?

To use the spec file to create an identical environment on the same machine or another machine: conda create --name myenv --file spec-file.txt To use the spec file to install its listed packages into an existing environment: conda install --name myenv --file spec-file.txt


2 Answers

Options from the Conda CLI

This is sort of what the --from-history flag is for, but not exactly. Instead of including exact build info for each package, it will include only what are called explicit specifications, i.e., the specifications that a user has explicitly requested via the CLI (e.g., conda install scipy=1.3.1). Have a try:

conda env export --from-history --name my_env > myenv.yml

This will only include versions if the user originally included versions during installation. Hence, creating a new environment is very likely not going to use the exact same versions and builds. On the other hand, if the user originally included additional constraints beyond version and build they will also be included (e.g., a channel specification conda install conda-forge::numpy will lead to conda-forge::numpy).

Another option worth noting is the --no-builds flag, which will export every package in the YAML, but leave out the build specifiers. These flags work in a mutually exclusive manner.


conda-minify

If this is not sufficient, then there is an external utility called conda-minify that offers some functionality to export an environment that is minimized based on a dependency tree rather than through the user's explicit specifications.

like image 184
merv Avatar answered Nov 13 '22 10:11

merv


Have a look at pipreqs. It creates a requirements.txt file only based on the imports that you are explicitely doing inside your project (and you even have a --no-pin option to ignore the version numbers). You can later use this file to create a conda environemnt via conda install --file requirements.txt.

However, if you're aiming for an evironments.yml file you have to create it manually. But that's just copy and paste from the clean requirements.txt. You only have to separate conda from "pip-only" installs.

like image 28
Peter Avatar answered Nov 13 '22 09:11

Peter