Preferred file line by line read idiom in Python

Question

I feel like almost every time I read a file in Python, what I want is:

with open("filename") as file_handle:
    for line in file_handle:
        #do something

Is this truly the preferred idiom? It mildly irritates me that this double indents all file reading logic. Is there a way to collapse this logic into one line or one layer?

Answer 1

For simple cases, yes, the two-level with and for is idiomatic.

For cases where the indentation becomes a problem, here as anywhere else in Python, the idiomatic solution is to find something to factor out into a function.

---

You can write wrappers to help this. For example, here's a simple way to solve some of the problems you use with for (e.g., even in the best case, the file sticks around after you finish the loop, until the end of the scope—which could be days later, or never, if the scope is a main event loop or a generator or something…):

def with_iter(iterable):
    with iterable:
        yield from iterable

for line in with_iter(open("filename")):
    # do something

for line in with_iter(open("other_filename")):
    # do something else

Of course it doesn't solve everything. (See this ActiveState recipe for more details.)

If you know that it does what you want, great. If you don't understand the differences… stick to what's idiomatic; it's idiomatic for a reason.

---

So, how do you refactor the code? The simplest way is often to turn the loop body into a function, so you can just use map or a comprehension:

def do_with_line(line):
    return line

with open("filename") as f:
    process = [do_with_line(line) for line in f]

But if the problem is that the code above or underneath the for is too deep, you'll have to refactor at a different level.

Answer 2

Yes, this is absolutely idiomatic Python.

You shouldn't be bothered too much by multiple levels of indentation. Certainly this is not the only way for it to happen, e.g.

if condition:
    for x in sequence:
        #do something with x

If the level of indentation becomes too great, it's time to refactor into multiple functions. One of the things I love most about Python is that it reduces the friction of breaking things up.

with open("filename") as file_handle:
    result = do_something(file_handle)