How do I make ipython aware of changes made to a selfwritten module?

Question

When trying a selfwritten module, it is likely to end with errors at the first few times.

But when fixing these errors, ipython does not seem to notice that.

Is there an ipython command to reload the new module? 'clear' does not do the trick. So far, the only thing that works is to 'exit' and start a new session. But this also means redoing everything I have done so far.

Or do I need to add something to the module that it kills all its internal variables after it has been run?

Example:

from mymodule import readCSVts
import pandas as pd
data = readCSVts('file.csv')

TypeError                                 Traceback (most recent call last)
<ipython-input-158-8f82f1a78260> in <module>()
----> 1 data = readCSVts('file.csv')

/home/me/path/to/mymodule.py in readCSVts(filename)
    194                 Cons_NaNs=hydroTS[(hydroTS.level.isnull())&(hydroTS.level.shift().isnull())&(hydroTS.level.shift(periods=2).isnull())]
    195                 #This is a pandas dataframe containing all rows with NaN
    196                 Cons_NaNs_count = len(Cons_NaNs)
    197                 Cons_NaNs_str = str(Cons_NaNs_count)
    198                 Cons_NaN_Name_joiner = [current_csv,';',Cons_NaNs]
--> 199                 Cons_NaN_Name_str = ''.join(Cons_NaN_Name_joiner)

TypeError: sequence item 2: expected string, DataFrame found

OK, that's easy. I made a typo in line 198, and wrote Cons_NaNs instead of Cons_NaNs_str, and thus I get the obvious error of trying to join a dataframe with a string.

But after fixing it in the mymodule.py file, I get the following (shortened) error:

    197                 Cons_NaNs_str = str(Cons_NaNs_count)
    198                 Cons_NaN_Name_joiner = [current_csv,';',Cons_NaNs_str]
--> 199                 Cons_NaN_Name_str = ''.join(Cons_NaN_Name_joiner)

TypeError: sequence item 2: expected string, DataFrame found

Looking at the traceback, ipython is well aware of the changes is made in the source file, it shows that I fixed the typo with the missing _str, but it still gives an error, that at the first look seems to be impossible. After running clear and reimporting everything, it shows the same behaviour.

So just to make sure that I did not make a stupid mistake somewhere along the way, I went trough my whole module step by step in ipython. And every variable that leads me to that point behaves as expected.

Cons_NaNs is a dataframe, Cons_NaNs_count is an integer and Cons_NaNs_str is a string.

So I exited ipython, restarted it and reimported everything and now it works.

But having to exit ipython sucks. Most of the times this means having to reimport dozens of things and doing a few dozen commands, to get to the point where I can actually test what I am currently working on.

Answer 1

There is an Ipython specific way, you can use set up autoreload:

In [1]: %load_ext autoreload

In [2]: %autoreload 2

In [3]: from foo import some_function

In [4]: some_function()
Out[4]: 42

In [5]: # open foo.py in an editor and change some_function to return 43

In [6]: some_function()
Out[6]: 43

The module was reloaded without reloading it explicitly, and the object imported with from foo import ... was also updated.

Usage The following magic commands are provided:

%autoreload

Reload all modules (except those excluded by %aimport) automatically now.

%autoreload 0

Disable automatic reloading.

%autoreload 1

Reload all modules imported with %aimport every time before executing the >Python code typed.

%autoreload 2

Reload all modules (except those excluded by %aimport) every time before executing the Python code typed.

%aimport

List modules which are to be automatically imported or not to be imported.

%aimport foo

Import module ‘foo’ and mark it to be autoreloaded for %autoreload 1

%aimport -foo

Mark module ‘foo’ to not be autoreloaded.

There is also dreload which will work for python2 and 3.

 dreload(module)

Answer 2

This is not just for ipython , but Python in general caches a module when it is first imported into sys.modules . So after the first import, whenever you try to import it, you would get the cached module object from sys.modules .

To make Python reload the module object without having to restart Python, so that changes done to the module are reflected, you should use reload() built-in function (Python 2.x) or importlib.reload() (Python 3.x).

Python 2.x -

<module> = reload(<module>)

Example -

import module
module = reload(module) #This requires the module as it is, not a string.

Python 3.x -

import importlib
<module> = importlib.reload(<module>)

Similar to Python 2.x example above, just use importlib.reload() instead of reload()