How to make imports / closures work from IPython's embed?

Question

I sometimes use embed at a certain point in a script to quickly flesh out some local functionality. Minimal example:

#!/usr/bin/env python

# ...

import IPython
IPython.embed()

Developing a local function often requires a new import. However, importing a module in the IPython session does not seem to work, when used in a function. For instance:

In [1]: import os

In [2]: def local_func(): return os.path.sep

In [3]: local_func()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-f0e5d4635432> in <module>()
----> 1 local_func()

<ipython-input-2-c530ce486a2b> in local_func()
----> 1 def local_func(): return os.path.sep

NameError: global name 'os' is not defined

This is rather confusing, especially since I can even use tab completion to write os.path.sep.

I noticed that the problem is even more fundamental: In general, functions created in the IPython embed session do not close over variables from the embed scope. For instance, this fails as well:

In [4]: x = 0

In [5]: def local_func(): return x

In [6]: local_func()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-f0e5d4635432> in <module>()
----> 1 local_func()

<ipython-input-5-2116e9532e5c> in local_func()
----> 1 def local_func(): return x

NameError: global name 'x' is not defined

Module names are probably just the most common thing to "close over"...

Is there any solution to this problem?

Update: The problem not only applies for closures, but also nested list comprehensions.

Disclaimer: I'll post an (unsatisfactory) answer to the question myself -- still hoping for a better solution though.

Answer 1

I also had the same problem. I used this trick to deal with the case when the embed() is called outside a function, so that globals() and locals() should be the same dictionary.

The simplest way is to call the following function after ipython is launched

ipy = get_ipython()
setattr(ipy.__class__, 'user_global_ns', property(lambda self: self.user_ns))

Another way is to subclass InteractiveShellEmbed

class InteractiveShellEmbedEnhanced(InteractiveShellEmbed):
    @property
    def user_global_ns(self):
        if getattr(self, 'embedded_outside_func', False):
            return self.user_ns
        else:
            return self.user_module.__dict__

    def init_frame(self, frame):
        if frame.f_code.co_name == '<module>':
            self.embedded_outside_func = True
        else:
            self.embedded_outside_func = False

and modify slightly the code of IPython.terminal.embed.embed() so that in it all InteractiveShellEmbed is changed to InteractiveShellEmbedEnhanced and call shell.init_frame(frame) after the line shell = InteractiveShellEmbed.instance(...).

This is based on the following observations:

• In an ipython session, we always have id(globals()) == id(ipy.user_module.__dict__) == id(ipy.user_global_ns) (user_global_ns is a class property of the super class of InteractiveShellEmbed, which returns ipy.user_module.__dict__)
• Also we have id(locals()) == id(ipy.user_ns)
• For normal ipython session, id(locals()) == id(globals())
• user_global_ns (a property) and user_ns (a dict) define the execution context
• In embedded ipython, ipy.user_module and ipy.user_ns are set in function ipy.__call__() and passed to ipy.mainloop(). They are not the same object since ipy.user_ns is constructed inside the functions.

If you are to launch ipython outside a function (like in a script), then it is safe to assume the globals() should be identical to locals().

With this setup, the following code should work while not working using the default embedded shell:

a=3
(lambda :a)()    # default behavior: name 'a' is not defined
import time
(lambda: time.time())()  # default behavior: name 'time' is not defined

(default behavior is due to a and time are not added to globals() and ipython does not make closures for local functions (the lambdas defined above) and insists to look up the variables in global scope. search closure in this page)

Answer 2

Update: Again only a work-around, but somewhat simpler: globals().update(locals())

---

I don't have a general solution, but at least a work-around: After defining a local function, it is possible to add the locals() of the session to the func_globals of the function just defined, e.g.:

In [1]: import os

In [2]: def local_func(): return os.path.sep

In [3]: local_func.func_globals.update(locals())

In [4]: local_func()
Out[4]: '/'

However, one should be aware that this is only a "manual closure" and will not work as a regular closure in cases like this:

In [1]: x = 1

In [2]: def local_func(): return x

In [3]: local_func.func_globals.update(locals())

In [4]: local_func()
Out[4]: 1

In [5]: x = 42

In [6]: local_func() # true closure would return 42
Out[6]: 1

In [7]: local_func.func_globals.update(locals()) # but need to update again

In [8]: local_func()
Out[8]: 42

At least it can solve the notorious global name '...' is not defined problem for imports.