Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

export conda environment without prefix variable which shows local path to executable

In order to improve reproducibly across my team I have exported a conda environment file in a newly created repository running conda env export > environment.yml.

The idea is for my colleagues to download the repo and run conda env create -f environment.yml and be ready to use the code.

However, some of my colleagues pointed out to me that at the bottom of the yaml file there is:

  - readline=7.0=hc1231fa_4
  - requests=2.18.4=py35h0d65e6b_1
  - setuptools=38.5.1=py35_0
  - six=1.11.0=py35h39a4c60_1
  - sqlite=3.22.0=h3efe00b_0
  - tk=8.6.7=h35a86e2_3
  - urllib3=1.22=py35he002d57_0
  - wheel=0.30.0=py35h5c0b906_1
  - xz=5.2.3=h0278029_2
  - zlib=1.2.11=hf3cbc9b_2
prefix: </path/to/your/anaconda/distribution>/envs/<env-name>/bin

the last line with prefix variable actually shows my machine name and unique path to miniconda installation.

When my colleagues tried to load the environment the script actually works fine for them, which means that the prefix variable is basically ignored by conda.

I used to work with virtualenvs and the idea that my absolute path goes into a file which is supposed to be shared to make things reproducible on different machines really confounds me.

So my question is: what is the prefix variable used for and is there a way to export a conda environment file without it?

like image 848
Giacomo Avatar asked Mar 08 '18 13:03

Giacomo


People also ask

How do you find the conda environment path?

Environments created with conda , by default, live in the envs/ folder of your miniconda3 (or anaconda3 ) directory the absolute path to which will look something the following: /Users/$USERNAME/miniconda3/envs or C:\Users\$USERNAME\Anaconda3 . You can see the location of your conda environments by running the command.

What is prefix in Yml file?

The + prefix is a dbt syntax feature, introduced in dbt v0. 17.0, which helps disambiguate between resource paths and configs in dbt_project. yml files. It is only compatible with dbt_project.yml files that use `config-version: 2`

Where does conda store environment variables?

The environments created by Conda is always located in /Users/.../anaconda3/envs/ . You may change the default location by using the following command but it is not encouraged.


1 Answers

This question was already addressed here, but without a real explanation on the role of prefix. At least there is a solution to exclude the prefix line programmatically.

It is not mentioned in the conda doc, except for the fact that conda env export --prefix PATH allows to specify a prefix. But note that --name and --prefix options are exclusive here.

If you have a look at conda's code, you'll see that conda create refers to cli_install.check_prefix(). And that install.py seems to indicate that there is a safety check on the environment name (extracted from the prefix) and on the full prefix path in order to ensure that no environment exists with the same name or path.

from conda/cli/install.py

def check_prefix(prefix, json=False):
    name = basename(prefix)
    error = None
    if name == ROOT_ENV_NAME:
        error = "'%s' is a reserved environment name" % name
    if exists(prefix):
        if isdir(prefix) and 'conda-meta' not in os.listdir(prefix):
            return None
        error = "prefix already exists: %s" % prefix

    if error:
        raise CondaValueError(error, json)

    if ' ' in prefix:
        stderrlog.warn("WARNING: A space was detected in your requested environment path\n"
                       "'%s'\n"
"Spaces in paths can sometimes be problematic." % prefix)

My guess is that this prefix in environment.yaml is part of a complex strategy to ensure conda will know where to create the environment. But it seems it is useless in most cases, and probably its presence is only due to the mechanistic link between --name and --prefix.

from conda.base.context.py Context()

# This block of code sets CONDA_PREFIX based on '-n' and '-p' flags, so that
# configuration can be properly loaded from those locations

EDIT:

The fact that prefix is not even mentioned in the "creating an environment file manually" section of the conda doc is comforting in the idea that this line is useless...

like image 189
FabienP Avatar answered Sep 21 '22 17:09

FabienP