Difference between iterating over a file-like and calling readline

Question

I always thought iterating over a file-like in Python would be equivalent to calling its readline method in a loop, but today I found a situation where that is not true. Specifically, I have a Popen'd process p where

list(itertools.takewhile(lambda x: x != "\n",
                         p.stdout))

hangs (presumably because p waits for input; both stdin and stdout are pipes to my Python process), while the following works:

list(itertools.takewhile(lambda x: x != "\n",
                         iter(p.stdout.readline, "")))

Can someone explain the difference?

Answer 1

The difference is purely in the implementation of iteration versus the readline method. File iteration reads in blocks (of 8 kilobytes, by default) and then splits up the buffer into lines as you consume them. The readline method, on the other hand, takes care never to read more than one line, and that means reading character by character. Reading in blocks is much more efficient, but it means you can't mix other operations on the file between reads. The expectation is that when you are iterating over the file, your intent is to read all lines sequentially and you will not be doing other operations on it. The readline method can't make that assumption.

As Sven Marnach hinted in his comment to your question, you can use iter(f.readline, '') to get an iterator that reads lines from the file without reading in blocks, at the cost of performance.