Using stacked conda environments

Question

I want to use multiple conda environments together. I have a huge conda environment containing a lot of packages (lets call it the monolith) which I use in all my projects and don't want to create again. I want to create a separate smaller conda environments for each project and work use it along with my huge monolith. So that I can keep the monolith clean and use for multiple projects safely. Following are a few things I think should be taken care of,

1. Update PATH, PYTHONPATH and LD_LIBRARY_PATH variables.
2. When installing a new package run a try dry run on all the environments in the stack and only then install it to the top environment. So that any version conflicts can be caught.
3. While executing the dry runs keep track of all the packages conda lists for installation. And when running the final install on top environment install only the intersection of packages listed in each of the dry run with --no-deps flag. This way we can avoid reinstallation of packages.

Would this approach work?

Answer 1

So the problem could be solved in two ways:

• The Cleaner Way

• The Clever Way

The Cleaner Way

You have the virtual environment monolith with you which you want to use in every project. For every project copy the virtual environment monolith with the project name and use it as the virtual environment.

The advantage for this way would be that we will have a clean and separate virtual environment to use. The cost for this way would be large space acquired by the same data, since you are copying monolith in every project.

The Clever Way

Create a copy of monolith virtual environment (only for safety). Make the folder containing the virtual environment packages a local git repo. The following command will be useful.

• git init

• git add .

• git commit -m"Master Project"

Now for every new project create a new branch using git checkout -b PROJECT_NAME and

Don't forget to switch to the branch you want to use. Most important when you are installing any package.

P.S : The Clever may or may not work depending on your system, I would suggest to go with cleaner way. Since the project domains will not be more than 6 or 7. (i.e one for ML, another for CV....)

Also, please comment which worked for you.

Answer 2

The question asks about a stacked environment, which is easily achieved with conda activate —-stack your_env.

An alternative could be to try docker’s union filesystem. Put your big base env in a docker image. Create a new image for your project that inherits from the base image. Now changes won’t affect the base env.

Or use UnionFS directly without docker. It’s not terribly hard but may require several steps to manage it.