Cloning generators in Python without Tee

Question

I am trying to clone a python generator object. I am bound by design so that I cannot make other changes, other than having a copy of the generator returned by a function and re-use it again. I am aware of the Itertools.tee() method, but the documentation says:

In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

How do I implement this? Is it saying to convert the generator to a list, copy that and convert both the lists to iterators/generators?

Answer 1

The docs are saying that you could achieve the same thing by materializing your iterable into a list then "copy" the iterable using the list as auxiliary storage, something to the effect of:

>>> def tee_two(iterable):
...     mem = list(iterable)
...     return iter(mem), iter(mem)
...
>>> en = enumerate('abc')
>>> next(en)
(0, 'a')
>>> it1, it2 = tee_two(en)
>>> for i, x in it1:
...     print(i, x)
...
1 b
2 c
>>> for i, x in it2:
...     print(i, x)
...
1 b
2 c

Of course, this requires materializing the rest of your iterator, and it is not memory efficient -- what if you have an infinite iterator? -- but if as the docs state if "one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee()" and potentially not much worse memory-wise.