Meaning of "OSError: telling position disabled by next() call" error?

Question

This is almost the same question as How to solve "OSError: telling position disabled by next() call". While the older question has received a few answers with useful workarounds, the meaning of the error is not clear. I wonder if anybody can comment on this.

I am learning Python and loosely following a tutorial. I entered the following interactively on Fedora 23:

$ python3
Python 3.4.3 (default, Aug  9 2016, 15:36:17)
[GCC 5.3.1 20160406 (Red Hat 5.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("myfile","r") as file:
...     for l in file:
...         print("Next line: \"{}\"".format(l))
...         print("Current position {:4d}".format(file.tell()))

myfile contains a few lines of text. The output:

Next line: "This is line number 0
"
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
OSError: telling position disabled by next() call

Googling for this error yields a whopping 6 results. The same happens with Python 3.6.4 on Cygwin on Windows 10.

Edit:

The tell() method for text files is documented as follows:

Return the current stream position as an opaque number. The number does not usually represent a number of bytes in the underlying binary storage.

"Opaque number" seems to indicate that I can't just print it. So, I replaced the second print() call with pos = file.tell(). Same result.

Answer 1

The message means exactly what it says: because you have called next() on the file, the use of tell() on that file has been disabled.

It might not look like you've called next, but the for loop calls it implicitly. A for loop:

for element in thing:
    do_stuff_with(element)

is syntactical sugar for

iterator = iter(thing) # the real implementation doesn't use a variable
while True:
    try:
        element = next(iterator) # here's the next() call
    except StopIteration:
        break
    do_stuff_with(element)

For a file, iter(file) returns the file, and the loop calls next on the file.

---

As for why calling next disables tell(), this is for efficiency. It only happens for text files (specifically io.TextIOWrapper), which have to do a bunch of extra work to support tell; turning off tell support lets them skip that work. The original commit message for the change that made next disable tell is "Speed up next() by disabling snapshot updating then.", indicating it's for efficiency.

For historical context, previous Python versions used a hidden buffer for next that tell and other file methods didn't account for, causing tell (and other file methods) to produce not-very-meaningful results during iteration over a file. The current IO implementation would be able to support tell() during iteration, but io.TextIOWrapper prevents such calls anyway. The historical incompatibility between next and other methods likely contributed to why it was considered reasonable to disable parts of file functionality during iteration.

---

You didn't ask for workarounds, but for the benefit of people who end up on this page looking for a workaround, I'll mention that

for line in iter(file.readline, ''):
    ...

will let you iterate over the lines of a text file without disabling tell. (You can use for line in iter(file.readline, b'') for binary files, but there's not much point, because the tell disabling mechanism isn't there for binary files.)

Answer 2

If your text file is too large, there are two solutions according to this answer:

1. Using file.readline() instead of next()

with open(path, mode) as file:
    while True:
        line = file.readline()
        if not line:
            break
        file.tell()

2. Using offset += len(line) instead of file.tell()

offset = 0
with open(path, mode) as file:
    for line in file:
        offset += len(line)