python3 file.readline EOF?

Question

I am having trouble determining when I have reached the end of a file in python with file.readline

fi = open('myfile.txt', 'r')
line = fi.readline()
if line == EOF:  //or something similar
    dosomething()

c = fp.read() if c is None: will not work because then I will loose data on the next line, and if a line only has a carriage return I will miss an empty line.

I have looked a dozens or related posts, and they all just use the inherent loops that just break when they are done. I am not looping so this doesn't work for me. Also I have file sizes in the GB with 100's of thousands of lines. A script could spend days processing a file. So I need to know how to tell when I am at the end of the file in python3. Any help is appreciated. Thank you!

Answer 1

I ran in to this same exact problem. My specific issue was iteration over two files, where the shorter one was only supposed to read a line on specific reads of the longer file.

As some mentioned here the natural pythonic way to iterate line by line is to, well, just iterate. My solution to stick with this 'naturalness' was to just utilize the iterator property of a file manually. Something like this:

with open('myfile') as lines:
    try:
        while True:                 #Just to fake a lot of readlines and hit the end
            current = next(lines)
    except StopIteration:
        print('EOF!')

You can of course embellish this with your own IOWrapper class, but this was enough for me. Just replace all calls to readline to calls of next, and don't forget to catch the StopIteration.

Answer 2

The simplest way to check whether you've reached EOF with fi.readline() is to check the truthiness of the return value;

line = fi.readline()
if not line:
    dosomething() # EOF reached

Reasoning

According to the official documentation

f.readline() reads a single line from the file; a newline character (\n) is left at the end of the string, and is only omitted on the last line of the file if the file doesn’t end in a newline. This makes the return value unambiguous; if f.readline() returns an empty string, the end of the file has been reached, while a blank line is represented by '\n', a string containing only a single newline.

and the only falsy string in python is the empty string ('').

Answer 3

You can use the output of the tell() function to determine if the last readline changed the current position of the stream.

fi = open('myfile.txt', 'r')
pos = fi.tell()

while (True):
    li = fi.readline()
    newpos = fi.tell()
    if newpos == pos:  # stream position hasn't changed -> EOF
        break
    else:
        pos = newpos

According to the Python Tutorial:

f.tell() returns an integer giving the file object’s current position in the file represented as number of bytes from the beginning of the file when in binary mode and an opaque number when in text mode.

...

In text files (those opened without a b in the mode string), only seeks relative to the beginning of the file are allowed (the exception being seeking to the very file end with seek(0, 2)) and the only valid offset values are those returned from the f.tell(), or zero.

Since the value returned from tell() can be used to seek(), they would have to be unique (even if we can't guarantee what they correspond to). Therefore, if the value of tell() before and after a readline() is unchanged, the stream position is unchanged, and the EOF has been reached (or some other I/O exception of course). Reading an empty line will read at least the newline and advance the stream position.