Jupyter / IPython: After editing a module, changes are not effective even with kernel restart

Question

I'm writing some code in a Jupyter notebook, and for some reason it is not recognizing when I've made changes to a module that the notebook imports.

I have the following cell at the top of my Jupyter notebook:

%load_ext autoreload
%autoreload 2

Based on this, my understanding is that this cell will force modules to reload upon execution of any code within the notebook. Meaning if I edit one of my imported modules, the edit should be reflected next time I run the code. Even without this cell though, I should be able to restart the module and run each cell from scratch to the same effect. Unfortunately this is not the case. No matter how many times I kill/restart the module, or even completely shut down Jupyter notebooks, the notebook will not recognize the changes I've made to the module(s) in question.

I'm working in a virtual environment, and so far the only solution I've found is to run a script that completely rebuilds the environment (including re-downloading packages, etc). This is a pain, and a much bigger waste of time than should be necessary.

This behavior is relatively new. I've been using Jupyter notebooks on this environment for months without such problems, but I can't pinpoint what I may have changed to cause this new behavior.

Any insight would be greatly appreciated.

Edit - Let me give a quick illustration of such a problem:

Say, in a module called foo.py I have a class:

class Foo(object):
    foo_bool = True

And in my notebook I have the cell:

from foo import Foo

Then later I add to foo.py:

class Foo(object):
    foo_bool = True


class Bar(object):
    bar_bool = False

And I now want to import both classes, so I change the notebook cell to:

from foo import Foo, Bar

Instead of properly importing both classes, I get something like:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-1d0ce6099263> in <module>()
      1 import some_module
      2 import some_other_module
----> 3 from foo import Foo, Bar
      4 from yet_another_module import Thing
      5 from the_last_module import TheLastThing

ImportError: cannot import name Bar

Thanks again for the help!

Answer 1

I have the similar problem. I found an answer in How do I make ipython aware of changes made to a selfwritten module? works.

It will also help in your case. According to that answer, in your context, the following code should solve your problem. (It is in Python 3. Python 2 has the similar solution. Please refer to the above link).

import importlib
import foo
from foo import Foo, Bar

foo = importlib.reload(foo)

I have no idea why the magic commands, %load_ext autoreload and %autoreload 2, are not working though. It will be helpful if someone can shed light on it.

Answer 2

Try to install your package with the --editable or -e flag,

pip install --editable <your-package>
# or
pip install --editable .

The issue you're experiencing could be due to you using pip install <your-package> or pip install . to install the module. This will always produce a copy of your source code and move it to the pip install directory. No matter how many times you import this module, it will always refer to the same "snapshot" of your code from when you installed it.

If instead you use pip install --editable <your-package> or pip install --editable . to install your package, the pip install of your source will refer to the actual directory you did the install from. When you then change code there, it should correctly be reflected in your notebook (assuming you're still using the magic commands; otherwise you'll need to use importlib or do a kernel restart to reimport the modules)

%load_ext autoreload
%autoreload 2