Error: module not found in Jupyter Notebook (sklearn)

Question

I have an issue importing scikit-learn in my Jupyter notebooks, and I am at a loss as to why this is not working. I do not recall having this issue before with other packages. I have seen several threads with people having similar problems with this specific module, but none of the proposed checks have solved my problem, nor revealed to me what might be wrong. I have tried to add as much info as possible below, to try and figure out why this refuses to work.

I use anaconda to manage my packages, and scikit-learn was installed as follows:

source activate python3
conda install scikit-learn
Fetching package metadata .............
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /Users/overaa/anaconda/envs/python3:
#
scikit-learn              0.19.1           py36hffbff8c_0  

Now, if I start a python session directly from the command line, import sklearn works as intended and the module is imported correctly. Doing the same in Jupyter results in

ModuleNotFoundError: No module named 'sklearn'

I have checked a number of things. First, making sure that my jupyter and jupyter-notebook point to the correct environment

which jupyter
/Users/user_name/anaconda/envs/python3/bin/jupyter
which jupyter-notebook
/Users/user_name/anaconda/envs/python3/bin/jupyter-notebook

So the notebook checks out.

Then from a command-line python session (in the same environment as where I launch my notebook):

>>> import sklearn
>>> sklearn.__file__
'/Users/user_name/anaconda/envs/python3/lib/python3.6/site_packages/sklearn/__init__.py'

So scikit-learn is also installed in the right environment.

Now comparing that I am running the same version;

From command line:

>>> print(sys.version)
3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

From Jupyter notebook:

print(sys.version)
3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:04:09
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

Comparing the packages path.

From command line:

import site
print(site.getsitepackages())
['/Users/user_name/anaconda/envs/python3/lib/python3.6/site-packages']

From Jupyter notebook:

import site
print(site.getsitepackages())
['/Users/user_name/anaconda/envs/python3/lib/python3.6/site-packages']

Similarly for my sys.path

command line:

>>> print(sys.path)
['', '/Users/user_name/anaconda/envs/python3/lib/python36.zip', 
'/Users/user_name/anaconda/envs/python3/lib/python3.6', 
'/Users/user_name/anaconda/envs/python3/lib/python3.6/lib-dynload', 
'/Users/user_name/anaconda/envs/python3/lib/python3.6/site-packages', 
'/Users/user_name/anaconda/envs/python3/lib/python3.6/site_packages/setuptools-27.2.0-py3.6.egg']

Jupyter:

print(sys.path)
['', '/Users/user_name/anaconda/envs/python36/lib/python36.zip', 
'/Users/user_name/anaconda/envs/python36/lib/python3.6', 
'/Users/user_name/anaconda/envs/python36/lib/python3.6/lib-dynload', 
'/Users/user_name/anaconda/envs/python36/lib/python3.6/site-packages', 
'/Users/user_name/anaconda/envs/python36/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg', 
'/Users/user_name/anaconda/envs/python36/lib/python3.6/site-packages/IPython/extensions', 
'/Users/user_name/.ipython']

So everything is the same, except that the directory containing the setuptools differ between the command-line version and the Jupyter version. I also noticed that scikit-learn seems to be installed in the site_packages subdir, rather than in the site-packages subdirectory, which is where Jupyter appear to be looking for modules. So I tried two things. First, adding the path to my JUPYTER_PATH in my .bash_profile before running my notebook.

export JUPYTER_PATH="/Users/user_name/anaconda/envs/python36/lib/python3.6/site-packages:$JUPYTER_PATH"

I also tried to add it to my sys.path directly in the jupyter notebook, prior to trying to import scikit-learn

sys.path.append("/Users/user_name/anaconda/envs/python3/lib/python3.6/site_packages")

Both to no avail, sklearn still refuses to import in my notebook.

Any ideas as to why scikit-learn is not getting imported into my Jupyter Notebook would be highly appreciated. Are there more things I can check? And how do I get this to work?

I have tried activating/deactivating my environment, restart the kernels etc. also without success.

EDIT: I have no problems importing other packages, like numpy or scipy

Answer 1

Turns out that removing the ~/Library/Jupyter/kernels folder and restarting my notebook did the trick. Even though it appears that the kernel in jupyter kernelspec list was what I expected. Nevertheless it is now working. It appears that it is not a sklearn exclusive issue. I basically followed what this person wrote.

Answer 2

Make sure you have both numpy and scipy installed - scikit-learn depends on them but does not install them for you.

Also, after installing them, restart your notebook server.