How to insert text at line and column position in a file?

Question

I would like to insert a string at a specific column of a specific line in a file.

Suppose I have a file file.txt

How was the English test?
How was the Math test?
How was the Chemistry test?
How was the test?

I would like to change the last line to say How was the History test? by adding the string History at line 4 column 13.

Currently I read in every line of the file and add the string to the specified position.

with open("file.txt", "r+") as f:
    # Read entire file
    lines = f.readlines()

    # Update line
    lino = 4 - 1
    colno = 13 -1
    lines[lino] = lines[lino][:colno] + "History " + lines[lino][colno:]

    # Rewrite file
    f.seek(0)
    for line in lines:
        f.write(line)
    f.truncate()
    f.close()

But I feel like I should be able to simply add the line to the file without having to read and rewrite the entire file.

Answer 1

This is possibly a duplicate of below SO thread

Fastest Way to Delete a Line from Large File in Python

In above it's a talk about delete, which is just a manipulation, and yours is more of a modification. So the code would get updated like below

def update(filename, lineno, column, text):
    fro = open(filename, "rb")

    current_line = 0
    while current_line < lineno - 1:
        fro.readline()
        current_line += 1

    seekpoint = fro.tell()
    frw = open(filename, "r+b")
    frw.seek(seekpoint, 0)

    # read the line we want to update
    line = fro.readline()
    chars = line[0: column-1] + text + line[column-1:]

    while chars:
        frw.writelines(chars)
        chars = fro.readline()

    fro.close()
    frw.truncate()
    frw.close()


if __name__ == "__main__":
    update("file.txt", 4, 13, "History ")

In a large file it make sense to not make modification till the lineno where the update needs to happen, Imagine you have file with 10K lines and update needs to happen at 9K, your code will load all 9K lines of data in memory unnecessarily. The code you have would work still but is not the optimal way of doing it

Answer 2

The function readlines() reads the entire file. But it doesn't have to. It actually reads from the current file cursor position to the end, which happens to be 0 right after opening. (To confirm this, try f.tell() right after with statement.) What if we started closer to the end of the file?

The way your code is written implies some prior knowledge of your file contents and layouts. Can you place any constraints on each line? For example, given your sample data, we might say that lines are guaranteed to be 27 bytes or less. Let's round that to 32 for "power of 2-ness" and try seeking backwards from the end of the file.

# note the "rb+"; need to open in binary mode, else seeking is strictly
# a "forward from 0" operation.  We need to be able to seek backwards
with open("file.txt", "rb+") as f:
    # caveat: if file is less than 32 bytes, this will throw
    # an exception.  The second parameter, 2, says "from end of file"
    f.seek(-32, 2)

    last = f.readlines()[-1].decode()

At which point the code has only read the last 32 bytes of the file.¹readlines() (at the byte level) will look for the line end byte (in Unix, \n or 0x0a or byte value 10), and return the before and after. Spelled out:

>>> last = f.readlines()
>>> print( last )
[b'hemistry test?\n', b'How was the test?']

>>> last = last[-1]
>>> print( last )
b'How was the test?'

Crucially, this works robustly under UTF-8 encoding by exploiting the UTF-8 property that ASCII byte values under 128 do not occur when encoding non-ASCII bytes. In other words, the exact byte \n (or 0x0a) only ever occurs as a newline and never as part of a character. If you are using a non-UTF-8 encoding, you will need to check if the code assumptions still hold.

Another note: 32 bytes is arbitrary given the example data. A more realistic and typical value might be 512, 1024, or 4096. Finally, to put it back to a working example for you:

with open("file.txt", "rb+") as f:
    # caveat: if file is less than 32 bytes, this will throw
    # an exception.  The second parameter, 2, says "from end of file"
    f.seek(-32, 2)

    # does *not* read while file, unless file is exactly 32 bytes.
    last = f.readlines()[-1]
    last_decoded = last.decode()

    # Update line
    colno = 13 -1
    last_decoded = last_decoded[:colno] + "History " + last_decoded[colno:]

    last_line_bytes = len( last )
    f.seek(-last_line_bytes, 2)
    f.write( last_decoded.encode() )
    f.truncate()

Note that there is no need for f.close(). The with statement handles that automatically.

¹ The pedantic will correctly note that the computer and OS will likely have read at least 512 bytes, if not 4096 bytes, relating to the on-disk or in-memory page size.