Cannot import packages installed in new Conda environment

Question

I am trying to make my own conda python environment on HPC server, and something very strange is happening.

The problem

After creating a new conda environment, it appears that python is not seeing itself in this environment, and using the base environment... Thus I cannot use packages installed in the new environment, but I can see the ones in the base environment...

Here is what I did

I install my environment as follows:

$ conda create -n niml pip python=3.6.5
$ source activate niml
(niml) $ conda install -c conda-forge luigi

and then I check my installed packages:

(niml) $ conda list

and here is what I get, very basic environment:

# Name                    Version                   Build  Channel
botocore                  1.10.61                    py_0    conda-forge
ca-certificates           2018.4.16                     0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
docutils                  0.14                     py36_0    conda-forge
jmespath                  0.9.3                      py_1    conda-forge
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 7.2.0                hdf63c60_3
libstdcxx-ng              7.2.0                hdf63c60_3
lockfile                  0.12.2                     py_1    conda-forge
luigi                     2.7.6                    py36_0    conda-forge
ncurses                   6.1                  hf484d3e_0
openssl                   1.0.2o                        0    conda-forge
pip                       10.0.1                   py36_0
pyparsing                 2.2.0                      py_1    conda-forge
python                    3.6.5                hc3d631a_2
python-daemon             2.1.2                    py36_0
python-dateutil           2.7.3                      py_0    conda-forge
readline                  7.0                  ha6073c6_4
setuptools                39.2.0                   py36_0
six                       1.11.0                   py36_1    conda-forge
sqlite                    3.24.0               h84994c4_0
tk                        8.6.7                hc745277_3
tornado                   4.5.3                    py36_0    conda-forge
wheel                     0.31.1                   py36_0
xz                        5.2.4                h14c3975_4
zlib                      1.2.11               ha838bed_2

then I try running python and import the package which I installed luigi, and it does not find it which results in the following error.

(niml) $ python

>>> import luigi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'luigi'

I also tried to import numpy which is in the base environment and NOT in the new environment, and it works.

>>> import numpy

The above works, which means python thinks it is running in the base environment.

Python in my own environment is being run

I checked which python is being executed, and it is indeed the one in the new environment niml:

>>> import sys
>>> sys.executable

'~/.conda/envs/niml/bin/python'

Also checked from the command line, and it is the python executable within the environment:

(niml) $ which python

~/.conda/envs/niml/bin/python

I am running on HPC

I have created anaconda environments hundreds of times before and never had this problem. Only difference is that I am using an HPC server, and thus I had to make something like this:

module load python/anaconda3

this is the only difference I see from my usual workflow which might be creating this problem...

Anyone else has seen this problem before and was able to solve it??

Answer 1

I was trying to solve a similar issue and solved this through virtualenv rather than using a conda environment. I believe there is a conflict between Anaconda and your machine in that both think they are controlling your new environment which setting up a new environment in virtualenv seemed to fix.

If it's helpful, here's how to set up an environment using virtualenv. Create a location for your new environment if you don't have one already:

mkdir ~/virtualenvironment

Set up your virtual environment:

virtualenv ~/virtualenvironment/niml/ --python=python3.6.5

Activate your environment:

source bin/activate

Make sure that you've installed whatever packages you need:

pip install luigi

Check that the package imports properly in python:

python
import luigi

To deactivate:

source deactivate