Add text to end of line without loading file

Question

I need to store information into a very big file, in form of many dictionaries. That's not so important, is just to say that I tried to first get all the data into these dictionaries and I run out of memory (~60Gb).

Fine, so I want to add the data in the file without actually loading it in memory, by doing a loop on the lines and attaching to each line a bit of text. Is that possible? if so, how?

Answer 1

Did you try any code yourself at all, what were your findings? You might go along the following approach:

with open('/tmp/bigfile.new', 'w') as output:
    with open('/tmp/bigfile', 'r') as input:
        while True:
            line = input.readline().strip()
            if not line:
                break
            line += ' Look ma, no hands!'
            print(line, file=output)

Except of course that instead of 'look ma no hands' you'd have your extra dictionary ;-)