Python file.tell gives wrong value location

Question

I am trying to extract a number of locations from an existing file using Python. This is my current code for extracting the locations:

    self.fh = open( fileName , "r+")
    p = re.compile('regGen regPorSnip begin')
    for line in self.fh :
        if ( p.search(line) ):
            self.porSnipStartFPtr = self.fh.tell()
            sys.stdout.write("found regPorSnip")

This snippet is repeated a number of times (less the file open) with different search values, and seems to work: I get the correct messages, and the variables have values.

However, using the code below, the first write location is wrong, while subsequent write locations are correct:

    self.fh.seek(self.rstSnipStartFPtr,0)
    self.fh.write(str);
    sys.stdout.write("writing %s" % str )
    self.rstSnipStartFPtr = self.fh.tell()

I have read that passing certain read/readline options to fh can cause an erroneous tell value because of Python's tendency to 'read ahead'. One suggestion I saw for avoiding this is to read the whole file and rewrite it, which isn't a very appealing solution in my application.

If i change the first code snippet to:

  for line in self.fh.read() :
        if ( p.search(line) ):
            self.porSnipStartFPtr = self.fh.tell()
            sys.stdout.write("found regPorSnip")

Then it appears that self.fh.read() is returning only characters and not an entire line. The search never matches. The same appears to hold true for self.fh.readline().

My conclusion is that fh.tell only returns valid file locations when queried after a write operation.

Is there a way to extract the accurate file location when reading/searching?

Thanks.

Answer 1

The cause is (rather obscurely) explained in the docs for a file object's next() method:

When a file is used as an iterator, typically in a for loop (for example, for line in f: print line), the next() method is called repeatedly. This method returns the next input line, or raises StopIteration when EOF is hit. In order to make a for loop the most efficient way of looping over the lines of a file (a very common operation), the next() method uses a hidden read-ahead buffer. As a consequence of using a read-ahead buffer, combining next() with other file methods (like readline()) does not work right. However, using seek() to reposition the file to an absolute position will flush the read-ahead buffer.

The values returned by tell() reflect how far this hidden read-ahead buffer has gotten, which will typically be up to a few thousand bytes beyond the characters your program has actually retrieved.

There's no portable way around this. If you need to mix tell() with reading lines, then use the file's readline() method instead. The tradeoff is that, in return for getting usable tell() results, iterating over a large file with readline() is typically significantly slower than using for line in file_object:.

Code

Concretely, change the loop to this:

line = self.fh.readline()
while line:
    if p.search(line):
        self.porSnipStartFPtr = self.fh.tell()
        sys.stdout.write("found regPorSnip")
    line = fh.readline()

I'm not sure that's what you really want, though: tell() is capturing the position of the start of the next line. If want the position of the start of the line, then you need to change the logic, like so:

pos = self.fh.tell()
line = self.fh.readline()
while line:
    if p.search(line):
        self.porSnipStartFPtr = pos
        sys.stdout.write("found regPorSnip")
    pos = self.fh.tell()
    line = fh.readline()

or do it with a "loop and a half":

while True:
    pos = self.fh.tell()
    line = self.fh.readline()
    if not line:
        break
    if p.search(line):
        self.porSnipStartFPtr = pos
        sys.stdout.write("found regPorSnip")