What does it mean to "consume" in Python? In an iterator?

Question

I have been working in Python for a few months now, and it has occurred to me that I often overlook vocabulary that escapes me at first glance, instead trying to get the gist of an idea. Now, looking back, I still find myself confused beyond belief at what the term consume refers to. My initial interest came from explanations of iterators which spoke of a value of an iterator being consumed. However, looking around, this does not seem to be commonplace in the Python lexicon. Or is it? Digging around here finds mostly references to Web Services, and one or two discussions on how to hide this or that result of a function.

I suppose then, to break down my ignorance into a few base points:

1. Does "consuming" do different things in different Pythonic contexts?
2. What happens to data when it is consumed, such as in iter()?
3. When a variable is assigned to an iterator's result-- the allegedly consumed piece of data-- does it no longer belong to the iterator?
4. Can you consume more than one value from an iterator object in a single call to the iterator?

I hope that makes some sort of sense. Note that this is not in reference to any particular need; I'm simply confused beyond rational plausibility.

EDIT: One more thing... does an iterated value (when called with next()) stay in memory?

Answer 1

Concerning 2.

In fact, we must distinguish two cases.

Remember what Greg Hewgill wrote:

An "iterator" is a single object that is responsible for creating some sequence of elements. This sequence might be elements of an existing list, or it might be something calculated, like prime numbers or the decimal digits of π.

First case:

the iterator calculates the object that it must produce when stimulated; that is to say, the produced object wasn't existing before the call of next() . Consequently, if a name is assigned to the object, this latter will survive; if not , the object will exist without being binded to a name in a namespace during a certain time, and then it will vanish in the memory, that is to say the bits it occupies will be used for another object later or sooner.

The second case

is when the iterator returns formerly existing objects belonging to a list, a tuple, a dictionary, etc.. In this case, each object produced by a next() had already a binding with a name. Then if the object is assigned to a new name when it "pops" out of the iterator, there will be two names binded to the object. And if the object is not assigned to a name, it will continue to be binded to only one name, what is sufficient to maintain the object alive.

In common:

Each time an object is produced by a call of an iterator, if no name is assigned to him, the only result of the operation is that the iterator has been "consumed". It's a manner to say that even if there is no permanent consequence after the production of an object, it has happened something that let a trace inside the iterator.

One speaks of consuming the iterator when a name is assigned to the object, too, however, I don't want to confuse.

Note:

In fact, in case of an object pre-existing in a list, say, it may be that it had no name. But the list holds a reference of every object it "contains"... In fact a list doesn't "contains" objects, but only references to objects... Well that goes beyond what I wanted to say.

.

Concerning 3

You should'nt write 3: "When a variable is assigned to ..."

The word variable is a pitfall in Python because it has an ambiguous signification. There are no variables in Python, in the common sense known in other langages, that is to say a « delimited portion of memory whose value can change ». There are only objects. The word variable is habitually used to mean an identifier. So it is a better practice to call it identifier, or name. This avoids confusion.

.

Concerning 4

I don't think that it's possible to obtain two returns from the iterator with only one call next()

Answer 2

The term "consume" is an informal term that refers to the way iterators work in Python. An "iterator" is a single object that is responsible for creating some sequence of elements. This sequence might be elements of an existing list, or it might be something calculated, like prime numbers or the decimal digits of π.

When a caller asks for the "next" item from an iterator, the iterator provides the next item and then changes its own state so it's ready to produce the next item after that. This usually is what you expect.

If I have a generator that produces the sequence of increasing prime numbers, the first time I call it I will get 2 in return. The next time, I'll get 3. If I then give you a reference to that generator, you'll call it (for what you think is the first time) and get 5. There isn't any way to "reset" the generator so it will start at 2 again, except by creating a completely new instance of the generator. In that situation, I could say that I have already consumed the first two items before giving you the generator.

Answer 3

I'm not a Python expert, but I can say this: Most of the time, consumption in programming is the opposite of production. You can classify some processes as producers, which create values; and others as consumers, which use the values created by the producers.

In the case of an iterator, an iterator is a producer which goes through an iterable object and "produces" each value, one at a time, in order. To "consume" data from an iterator simply means to use it.

Answer 4

Iterators are just objects that support the methods __iter__ and next. The general use case for iterators is to loop over them, where each time through the loop the result of iterator.next() will be assigned to a variable.

With this in mind, a call to iterator.next() could be referred to as "consuming a value" because in general a call to next changes the state of the iterator, and there is no way to return to that previous state.

However, there is nothing preventing an iterator from returning the same value repeatedly, or even providing a way to roll back to a previous state. In those cases using the word "consume" may not be as applicable.

As far as what happens to the data that is returned by the iterator's next method, it is completely dependent on the implementation of the iterator. Generators tend to discard the results that they yield, but if a container is also an iterator then the data returned when next() is called will still exist in the container object.