What is the difference between 'in fp' and 'in fp.readlines()'?

Question

What is the difference between for line in fp and for line in fp.readlines()?

with open(filename, 'r') as fp :
    for line in fp.readlines() :

#AND

with open(filename, 'r') as fp :
    for line in fp :

Answer 1

file.readlines() “[reads] and [returns] a list of lines from the stream.” So what you get back is a list of every line. As such, the whole file is read into the memory and then split into lines.

The documentation already says this:

Note that it’s already possible to iterate on file objects using for line in file: ... without calling file.readlines().

So unless you have an actual need to get all lines as a list, don’t use readlines. Instead iterate over the file directly, because IOBase, which is the base type for all file handlers, implements the iterator protocol:

IOBase (and its subclasses) supports the iterator protocol, meaning that an IOBase object can be iterated over yielding the lines in a stream. Lines are defined slightly differently depending on whether the stream is a binary stream (yielding bytes), or a text stream (yielding character strings). See readline() below.

Using the iterator protocol has the benefit that the file will not be read completely into the memory. Instead, the file stream will be consumed iteratively and give you one line after another without having all the other contents of the file in the memory. So this works very well even for very large files.

Answer 2

fp - is the file object itself , you can iterate over them to get the lines in the file.

Example -

>>> f = open('test.csv','r')
>>> f
<_io.TextIOWrapper name='test.csv' mode='r' encoding='cp1252'>

You can only iterate over them , you cannot access a certain line in the file directly without using seek() or such function.

fp.readlines() - this returns the list of all lines in the file, when you iterate over this , you are iterating over the list of lines.

Example -

>>> f = open('test.csv','r')
>>> lines = f.readlines()
>>> lines
['order_number,sku,options\n', '500,GK-01,black\n', '499,GK-05,black\n', ',,silver\n', ',,orange\n', ',,black\n', ',,blue']

Here , you can get the 2nd line in the file using lines[1] , etc.

Usually if the requirement is to just iterate over the lines in the file, its better to use the file directly, since creating a list of lines and then iterating over them would cause unnecessary overhead.