When does Python decode a byte string while reading a file?

Question

I have a text file with keywords and I use

with open('filename.txt','r') as file:
    list_of_words = [x.strip('\n') for x in file.readlines()

I get a: UnicodeDecodeError : 'ascii' codec can't decode byte 0xc4 in position 5595: ordinal not in range(128) on line 2

I understand the error. I don't understand why is it on line 2.

According to python docs: https://docs.python.org/3/library/functions.html#open

In text mode (the default, or when 't' is included in the mode argument), the contents of the file are returned as str, the bytes having been first decoded using a platform-dependent encoding or using the specified encoding if given.

This means that while opening the file, the decoding process happens while opening the file and returned in 'file' variable.

Why do I get the error on line 2 then?

Answer 1

You seem to be confusing the file object returned by the open() call, for the actual process of reading from a file object.

Python decodes the contents of the file, as you read it. Opening a file object doesn't read any data from the file, it just creates a file object. No data is read from the file at that point, there are no bytes for Python to process yet.

In line 2 you actually read from the file, using the file.readlines() method. It is that method that tells the file object to fetch data from the filesystem (bytes) and decode those bytes. Only then can Python know that the data cannot actually be decoded as ASCII.

Answer 2

Opening the file doesn't actually examine the contents. Only when some of the file data is returned via one of its several methods that perform a read can the contents be decoded.