Accessing iterator in 'for in' loop

Question

From my understanding, when code like the following is run:

for i in MyObject:
    print(i)

MyObject's __iter__ function is run, and the for loop uses the iterator it returns to run the loop.

is it possible to access this iterator object mid-loop? Is it a hidden local variable, or something like that?

I would like to do the following:

for i in MyObject:
    blah = forloopiterator()
    modify_blah(blah)
    print(i)

I want to do this because I am building a debugger, and I need to modify the iterator after it has been instantiated (adding an object to be iterated during this loop, mid-execution). I am aware that this a hack and should not be done conventionally. Modifying MyObject.items (which is what the iterator is iterating over) directly doesn't work, sicne the iterator only evaluates once. So I need to modify the iterator directly.

Answer 1

It is possible to do what you want to do, as long as you're willing to rely on multiple undocumented internals of your Python interpreter (in my case, CPython 3.7)—but it isn't going to do you any good.

---

The iterator is not exposed to locals, or anywhere else (not even to a debugger). But as pointed out by Patrick Haugh, you can get at it indirectly, via get_referrers. For example:

for ref in gc.get_referrers(seq):
    if isinstance(ref, collections.abc.Iterator):
        break
else:
    raise RuntimeError('Oops')

Of course if you have two different iterators to the same list, I don't know if there's any way you can decide between them, but let's ignore that problem.

---

Now, what do you do with this? You've got an iterator over seq, and… now what? You can't replace it with something useful, like an itertools.chain(seq, [1, 2, 3]). There's no public API for mutating list, set, etc. iterators, much less arbitrary iterators.

if you happen to know it's a list iterator… well, the CPython 3.x listiterator does happen to be mutable. The way they're pickled is by creating an empty iterator and calling __setstate__ with a reference to a list and an index:

>>> print(ref.__reduce__())
(<function iter>, ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9],), 7)
>>> ref.__setstate__(3) # resets the iterator to index 3 instead of 7
>>> ref.__reduce__()[1][0].append(10) # adds another value

But this is all kind of silly, because you could get the same effect by just mutating the original list. In fact:

>>> ref.__reduce__()[1][0] is seq
True

So:

lst = list(range(10))
for elem in lst:
  print(elem, end=' ')
  if elem % 2:
    lst.append(elem * 2)
print()

… will print out:

0 1 2 3 4 5 6 7 8 9 2 6 10 14 18 

… without having to monkey with the iterator at all.

---

You can't do the same thing with a set.

Mutating a set while you're in the middle of iterating it will affect the iterator, just as mutating a list will—but what it does is indeterminate. After all, sets have arbitrary order, which is only guaranteed to be consistent as long as you don't add or delete. What happens if you add or delete in the middle? You may get a whole different order, meaning you may end up repeating elements you already iterated, and missing ones you never saw. Python implies that this should be illegal in any implementation, and CPython does actually check it:

s = set(range(10))
for elem in s:
  print(elem, end=' ')
  if elem % 2:
    s.add(elem * 2)
print()

This will just immediately raise:

RuntimeError: Set changed size during iteration

So, what happens if we use the same trick to go behind Python's back, find the set_iterator, and try to change it?

s = {1, 2, 3}
for elem in s:
    print(elem)
    for ref in gc.get_referrers(seq):
        if isinstance(ref, collections.abc.Iterator):
            break
    else:
        raise RuntimeError('Oops')
    print(ref.__reduce__)

What you'll see in this case will be something like:

2
(<function iter>, ([1, 3],))
1
(<function iter>, ([3],))
3
(<function iter>, ([],))

In other words, when you pickle a set_iterator, it creates a list of the remaining elements, and gives you back instructions to build a new listiterator out of that list. Mutating that temporary list obviously has no useful effect.

---

What about a tuple? Obviously you can't just mutate the tuple itself, because tuples are immutable. But what about the iterator?

Under the covers, in CPython, tuple_iterator shares the same structure and code as listiterator (as does the iterator type that you get from calling iter on an "old-style sequence" type that defines __len__ and __getitem__ but not __iter__). So, you can do the exact same trick to get at the iterator, and toreduce` it.

But once you do, ref.__reduce__()[1][0] is seq is going to be true again—in other words, it's a tuple, the same tuple you already had, and still immutable.