Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python's itertools product memory consumption

The documentation says that the cartesian product function

the actual implementation does not build up intermediate results in memory.

How can that be possible with generators? Can somebody show me an example with a bounded memory consumption for 2 generators?

like image 548
fulmicoton Avatar asked May 23 '12 12:05

fulmicoton


People also ask

Is Itertools faster than for loops?

That being said, the iterators from itertools are often significantly faster than regular iteration from a standard Python for loop.

What is product in Itertools in Python?

product() is used to find the cartesian product from the given iterator, output is lexicographic ordered. The itertools. product() can used in two different ways: itertools. product(*iterables, repeat=1):

What is Islice in Python?

islice() functionThis iterator selectively prints the values mentioned in its iterable container passed as an argument. Syntax: islice(iterable, start, stop, step)

What does Itertools Zip_longest return?

zip_longest() This iterator falls under the category of Terminating Iterators. It prints the values of iterables alternatively in sequence.


1 Answers

Looking at the module's source code, itertools.product() actually converts every argument to a tuple:

// product_new() in itertoolsmodule.c
for (i=0; i < nargs ; ++i) {
    PyObject *item = PyTuple_GET_ITEM(args, i);
    PyObject *pool = PySequence_Tuple(item); //<==== Call tuple(arg)
    if (pool == NULL)
        goto error;
    PyTuple_SET_ITEM(pools, i, pool);
    indices[i] = 0;
}

In other words, itertools.product()'s memory consumption appears to be linear in the size of the input arguments.

like image 187
NPE Avatar answered Sep 30 '22 02:09

NPE