Installing tensorflow in cloned conda environment breaks conda environment it was cloned from

Question

I just cloned a working miniconda environment carnd-term1-gpu that had tensorflow with gpu working using tensorflow version '0.12.1'. The environment had a lot of other python packages installed. I wanted to upgrade to the latest version of tensorflow so I did the following based on tensorflow Ubuntu:

1) cloned the existing working environment as tflow

conda create --name tflow --clone carnd-term1-gpu

This completed successfully.

2) source activate tflow and install using pip

source activate tflow
(tflow) xx@pc:~$ pip install --ignore-installed --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.1.0-cp35-cp35m-linux_x86_64.whl

Now I try to run the old environment assuming that it should not be changed.

source activate carnd-term1-gpu
ipython

Then from ipython prompt type import numpy as np and get following:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-4ee716103900> in <module>()
----> 1 import numpy as np

/home/ai/miniconda3/envs/carnd-term1-gpu/lib/python3.5/site-packages/numpy/__init__.py in <module>()
    140         return loader(*packages, **options)
    141 
--> 142     from . import add_newdocs
    143     __all__ = ['add_newdocs',
    144                'ModuleDeprecationWarning',

/home/ai/miniconda3/envs/carnd-term1-gpu/lib/python3.5/site-packages/numpy/add_newdocs.py in <module>()
     11 from __future__ import division, absolute_import, print_function
     12 
---> 13 from numpy.lib import add_newdoc
     14 
     15 ###############################################################################

/home/ai/miniconda3/envs/carnd-term1-gpu/lib/python3.5/site-packages/numpy/lib/__init__.py in <module>()
      6 from numpy.version import version as __version__
      7 
----> 8 from .type_check import *
      9 from .index_tricks import *
     10 from .function_base import *

/home/ai/miniconda3/envs/carnd-term1-gpu/lib/python3.5/site-packages/numpy/lib/type_check.py in <module>()
      9            'common_type']
     10 
---> 11 import numpy.core.numeric as _nx
     12 from numpy.core.numeric import asarray, asanyarray, array, isnan, \
     13                 obj2sctype, zeros

/home/ai/miniconda3/envs/carnd-term1-gpu/lib/python3.5/site-packages/numpy/core/__init__.py in <module>()
     51 from . import shape_base
     52 from .shape_base import *
---> 53 from . import einsumfunc
     54 from .einsumfunc import *
     55 del nt

ImportError: cannot import name 'einsumfunc'

I thought I was protected by cloning a separate environment and only changing that environment, however it seems that the install in one environment changed behavior in the other environment. Exactly what I thought I was avoiding! What went wrong and how do I fix this? The new environment seems to work fine so far.

I checked the file dates in both environments. The carnd-term1-gpu environment had files that were dated the same day that I did the upgrade in the tflow environment. Any ideas on how that could happen?

I tried adding the missing file einsumfunc.py to my carnd-term1-gpu environment and started Ipython again. This time when I did import numpy as np a different file could not be imported. So it looks like my carnd-term1-gpu environment is corrupted.

Comparing conda list for each environment and looking only at numpy, I see the following: For tflow environment:

numpy                     1.11.3                    <pip>
numpy                     1.12.1                    <pip>
numpy                     1.11.3          py35_blas_openblas_200  [blas_openblas]  conda-forge

And for carnd-term1-gpu environment:

numpy                     1.11.3                    <pip>
numpy                     1.11.3          py35_blas_openblas_200  [blas_openblas]  conda-forge

I then looked at revisions using conda list --revisions. Both environments only show a single revision of rev 0. Likewise looking at conda-meta/history only shows the original creation dates of Jan. 13th for carnd-term1-gpu and May 9 for tflow. So the combination of this version of pip:

pip                       9.0.1                    py35_0    conda-forge

combined with this wheel:

https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.1.0-cp35-cp35m-linux_x86_64.whl

somehow corrupted the environment carnd-term1-gpu. This seems like either a bug or I somehow got a corrupted wheel from the tensorflow site? If it is the later, how could it corrupt a different environment than the one installed from? Note version of pip was same in both environments.

I did the following to investigate what files changed on the date that I did the install.

find /home/ai/miniconda3/envs/carnd-term1-gpu/  -type f -newermt 2017-05-09 -ls | wc -l

This showed 669 files were created/modified. Most of these were in /site-packages/numpy/, but six.py, pyparsing.py,/setuptools/,/pkg_resources/,easy_install.py,/werkzeug/ were also affected.

conda --version is 4.2.12

I tried the following to get my old environment going again.

conda env export > environment.yml
conda env create -f environment.yml -n sdc-gpu

This resulted in following errors:

Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
  File "/home/ai/miniconda3/envs/sdc-gpu/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/home/ai/miniconda3/envs/sdc-gpu/lib/python3.5/site-packages/setuptools/__init__.py", line 12, in <module>
    import setuptools.version
  File "/home/ai/miniconda3/envs/sdc-gpu/lib/python3.5/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/home/ai/miniconda3/envs/sdc-gpu/lib/python3.5/site-packages/pkg_resources/__init__.py", line 70, in <module>
    import packaging.version
ImportError: No module named 'packaging'

CondaValueError: Value error: pip returned an error.

Answer 1

I just got the following from continuum; "To keep this from happening, you'll have to use the --copy flag with your clone operation. Core to conda's design is extensive use of hard links. This exact issue is one of the biggest pitfalls."

So if I would have done the following when I first cloned the environment, I could have avoided corrupting the old one:

conda create --name tflow --copy --clone carnd-term1-gpu

The option --copy Install all packages using copies instead of hard- or soft-link‐ing will prevent pip from being able to overwrite files.

Some ongoing discussion about this and how it might get addressed in the future is here: conda pip breaks

The only option I have for recovering is to reinstall each damaged package. Beware when using pip with conda...