Debug generators and iterators

Question

How do I debug code which uses generators and iterators? I found that adding a for loop with print statements consumes the generator/iterator and therefore breaks the rest of the code. Is it possible to inspect the "contents" of a generator/iterator without consuming the elements?

More specifically, I have something like this:

result = map(func, x)

Now I want to see what result is. I also want to see the value returned by applying a function to each element in result. In my actual code, I am getting the element in result which gives the minimum value of this function:

best = min(result, key=my_key)

Now min() is very handy, but I am getting incorrect behavior and need to figure out why. What are some tools I can use to debug something like this?

p.s. I am using PyCharm. I am pretty comfortable with the interactive debugger but still cannot figure out how to view everything that is going on here.

Answer 1

If you want to print the values of the function arguments, and the values returned, you could use a wrapper that wraps the original function and prints the values passed into the func or my_key, and their return values. Something like

def debug_func(func):
    @functools.wraps(func)
    def wrapper(*a, **kw):
        print('Arguments', a, kw)
        rv = func(*a, **kw)
        print('Return value', repr(rv))
        return rv
    return wrapper

An usage example:

>>> list(map(debug_func(len), ['foo', 'bar', 'foobar']))
Arguments ('foo',) {}
Return value 3
Arguments ('bar',) {}
Return value 3
Arguments ('foobar',) {}
Return value 6
[3, 3, 6]

Or

>>> min(['foo', 'bar', 'foobar'], key=debug_func(len))
Arguments ('foo',) {}
Return value 3
Arguments ('bar',) {}
Return value 3
Arguments ('foobar',) {}
Return value 6
'foo'

Similar approach could be used for iterators too:

def debug_iter(iterator):
    for value in iterator:
         print('Iterator yielded', repr(value))
         yield value

Usage:

>>> [i for i in debug_iter(i ** 2 for i in range(5)) if i % 2]
Iterator yielded 0
Iterator yielded 1
Iterator yielded 4
Iterator yielded 9
Iterator yielded 16
[1, 9]

---

Debugging on the fly (wrapping it in) inside pdb is possible, though tedious. This can work:

(Pdb) !import code; code.interact(local=vars())

will enter an embedded Python shell where you can provide the debug function, for example

def debug_iter(iterator):
    for value in iterator:
         print('Iterator yielded', repr(value))
         yield value

and then you should be able to do Ctrl+D to exit to pdb again, and do the wrapping there:

my_iter = debug_iter(my_iter)