Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how itertools.tee works, can type 'itertools.tee' be duplicated in order to save it's "status"?

Below are some tests about itertools.tee:

    li = [x for x in range(10)]
    ite = iter(li)
==================================================
    it = itertools.tee(ite, 5)
    >>> type(ite)
    <type 'listiterator'>
    >>> type(it)
    <type 'tuple'>
    >>> type(it[0])
    <type 'itertools.tee'>
    >>> 

    >>> list(ite)
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[0])          # here I got nothing after 'list(ite)', why?
    []
    >>> list(it[1])
    []
====================play again===================
    >>> ite = iter(li)
    it = itertools.tee(ite, 5)
    >>> list(it[1])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[2])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[3])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[4])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(ite)
    []                       # why I got nothing? and why below line still have the data?   
    >>> list(it[0])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[0])
    []
====================play again===================    
    >>> ite = iter(li)
    itt = itertools.tee(it[0], 5)    # tee the iter's tee[0].
    >>> list(itt[0])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(itt[1])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(it[0])
    []                               # why this has no data?
    >>> list(it[1])
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> list(ite)
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]  

My question is

  1. How does tee work and why sometimes the original iter 'has data' and other time it doesn't?
  2. Can I keep an iter deep copy as a "status seed" to keep the raw iterator status and tee it to use later?
  3. Can I swap 2 iters or 2 itertools.tee?

Thanks!

like image 605
user478514 Avatar asked Oct 18 '10 07:10

user478514


People also ask

How does Itertools tee work?

The tee() function can be used to create any number of independent iterators from a single iterable. It takes two arguments: the first is an iterable inputs , and the second is the number n of independent iterators over inputs to return (by default, n is set to 2). The iterators are returned in a tuple of length n .

What is chain in Itertools in Python?

It is a function that takes a series of iterables and returns one iterable. It groups all the iterables together and produces a single iterable as output.


1 Answers

tee takes over the original iterator; once you tee an iterator, discard the original iterator since the tee owns it (unless you really know what you're doing).

You can make a copy of a tee with the copy module:

import copy, itertools
it = [1,2,3,4]
a, b = itertools.tee(it)
c = copy.copy(a)

... or by calling a.__copy__().

Beware that tee works by keeping track of all of the iterated values that have been consumed from the original iterator, which may still be consumed by copies.

For example,

a = [1,2,3,4]
b, c = itertools.tee(a)
next(b)

At this point, the tee object underlying b and c has read one value, 1. It's storing that in memory, since it has to remember it for when c is iterated. It has to keep every value in memory until it's consumed by all copies of the tee.

The consequence of this is that you need to be careful with "saving state" by copying a tee. If you don't actually consume any values from the "saved state" tee, you're going to cause the tee to keep every value returned by the iterator in memory forever (until the copied tee is discarded and collected).

like image 77
Glenn Maynard Avatar answered Sep 20 '22 06:09

Glenn Maynard