Conda importing one environment.yml into another

Question

Consider the situation where you're importing a Python submodule with dependencies into a project with its own dependencies. Say that the submodule has its environment.yml file and the project itself has an environment.yml file.

Is there a way to create an environment incorporating both of these environment specifications, and if so, how would you go about doing this? (Or is there a better/more preferred way to handle this situation?)

Answer 1

I'm guessing you're the one developing both the submodule and the project since the dependencies are still in the environment.yml files.

Option 1: Update Project Environment using Submodule's environment.yml

This unfortunately might be your only option if your submodule has dependencies on conda packages.

# First create the project environment
$ conda env create --force -f project_environment.yml

# Then update with submodule dependencies
$ conda env update -n project-env-name --file submodule_environment.yml

This is less than ideal since the basic expectation is that your imported libraries come with their own dependencies.

Option 2: Put dependencies into respective requirements.txt files

This is applicable only if the submodule dependencies can be installed from PyPi via pip. First put the dependencies of the project and submodule into their respective requirements.txt files.

Then restructure the environment.yml files to look the following:

submodule_environment.yml

name: submodule-env-name
channels:
  - defaults
dependencies:
- python=3.6.3             # no conda dependencies
- pip:
    - -r requirements.txt  # <--- submodule dependencies

project_environment.yml

name: project-env-name
channels:
  - defaults
dependencies:
- python=3.6.3
- pip:
    - -r requirements.txt                    # <--- project dependencies
    - -r project/submodule/requirements.txt  # <--- submodule dependencies

In this way you can ignore the submodule_environment.yml file altogether and then create the project environment with a single command.

$ conda env create --force -f project_environment.yml

This approach will not work if your submodule has dependencies on conda packages. If it does then Option 1 is your best option.

Option 3: Package the Submodule (Ideal)

Assuming the submodule has no conda dependencies then it would be ideal to just make a separate package out of the submodule. Create a setup.py and put all the dependencies into the install_requires field. Here's a template of how the setup.py file should look like.

Once it's packaged, you can do the following:

• Install it locally using:
  • pip install .
• Upload to github or bitbucket and install it using:
  • pip install git+https://github.com/username/submodule.git --upgrade
• Upload to github or bitbucket and add the following to requirements.txt or environment.yml under pip:
  • git+https://github.com/username/submodule.git#egg=submodule