iPython debugger raises `NameError: name ... is not defined`

Question

I can't understand the following exception that is raised in this Python debugger session:

(Pdb) p [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined
(Pdb) [move for move in move_values]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) max_value
0.5
(Pdb) (0.5, (0, 2))[0] == max_value
True
(Pdb) [move for move in move_values if move[0] == 0.5]
[(0.5, (0, 0)), (0.5, (0, 1)), (0.5, (0, 2)), (0.5, (1, 0)), (0.5, (1, 1)), (0.5, (1, 2)), (0.5, (2, 0)), (0.5, (2, 1)), (0.5, (2, 2))]
(Pdb) [move for move in move_values if move[0] == max_value]
*** NameError: name 'max_value' is not defined

Why is it sometimes telling me max_value is not defined and other times not?

Incidentally, this is the code immediately prior to the debugger starting:

max_value = max(move_values)[0]
best_moves = [move for move in move_values if move[0] == max_value]
import pdb; pdb.set_trace()

I am using Python 3.6 running in PyCharm.

AMENDED UPDATE:

After more testing it appears that local variables are not visible within list comprehensions within a pdb session when I do the following from an iPython REPL or in PyCharm:

$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):
(Pdb) x = 1; [x for i in range(3)]
*** NameError: name 'x' is not defined

But in a regular Python REPL it works:

$ python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb; pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) x = 1; [x for i in range(3)]
[1, 1, 1]

I tested above with versions 3.4, 3.5, 3.6 so it does not appear to be version dependent.

UPDATE 2

Please note, the above test ('AMENDED UPDATE') is problematic because it uses import pdb; pdb.set_trace() in the interactive REPL.

Also, the original problem is not limited to iPython.

See answer by user2357112 below for a comprehensive explanation of what is going on here.

Sorry if I caused any confusion!

Answer 1

You've got two core problems here. The first is that (when calling pdb.set_trace() interactively in IPython) you're debugging IPython's guts instead of the scope you wanted. The second is that list comprehension scoping rules interact badly with cases where the variables present in enclosing scopes can't be determined statically, such as in debuggers or class bodies.

The first problem pretty much only happens when typing pdb.set_trace() into an IPython interactive prompt, which isn't a very useful thing to do, so the simplest way to avoid the problem is to just not do that. If you want to do it anyway, you can enter the r command a few times until pdb says you're out of IPython's guts. (Don't overshoot, or you'll end up in a different part of IPython's guts.)

The second problem is an essentially unavoidable interaction of heavily entrenched language design decisions. Unfortunately, it's unlikely to go away. List comprehensions in a debugger only work in a global scope, not while debugging a function. If you want to build a list while debugging a function, the easiest way is probably to use the interact command and write a for loop.

---

Here's the full combination of effects going into this.

1. pdb.set_trace() triggers pdb on the next trace event, not at the point where pdb.set_trace() is called.

The trace function mechanism used by pdb and other Python debuggers only triggers on certain specific events, and "when a trace function is set" is unfortunately not one of those events. Normally, the next event is either a 'line' event for the next line or a 'return' event for the end of the current code object's execution, but that's not what happens here.

2. IPython sets a displayhook to customize expression statement handling.

The mechanism Python uses to display the result of expression statements is sys.displayhook. When you do 1+2 at the interactive prompt:

>>> 1+2
3

sys.displayhook is what prints the 3 instead of discarding it. It also sets _. When the result of an expression statement is None, such as with the expression pdb.set_trace(), sys.displayhook does nothing, but it's still called.

IPython replaces sys.displayhook with its own custom handler, responsible for printing the Out[n]: thingy, for setting entries in the Out record, for calling IPython custom pretty-printing, and for all sorts of other IPython conveniences. For our purposes, the important thing is that IPython's displayhook is written in Python, so the next trace event is a 'call' event for the displayhook.

pdb starts debugging inside IPython's displayhook.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):

3. List comprehensions create a new scope.

People didn't like how list comprehensions leaked the loop variable into the containing scope in Python 2, so list comprehensions get their own scope in Python 3.

4. pdb uses eval, which interacts really poorly with closure variables.

Python's closure variable mechanism relies on static scope analysis that's completely incompatible with how eval works. Thus, new scopes created inside eval have no access to closure variables; they can only access globals.

---

Putting it all together, in IPython, you end up debugging the IPython displayhook instead of the scope you're running interactive code in. Since you're inside IPython's displayhook, your x = 1 assignment goes into the displayhook's locals. The subsequent list comprehension would need access to the displayhook's locals to access x, but that would be a closure variable to the list comprehension, which doesn't work with eval.

Outside of IPython, sys.displayhook is written in C, so pdb can't enter it, and there's no 'call' event for it. You end up debugging the scope you intended to debug. Since you're in a global scope, x = 1 goes in globals, and the list comprehension can access it.

You would have seen the same effect if you tried to run x = 1; [x for i in range(3)] while debugging any ordinary function.