Efficient way to delete lines from a file in python (and keep the same file name)?

Question

My program is keeping a log for the user. If the log ever gets bigger than a set amount, I want to delete the first 20% of lines.

From similar questions, I've seen suggestions to do read in the old file, and write out all the lines I want to keep into a new file. However, my files might be too large to be constantly reading them in, and using that method wouldn't let me keep the same file name.

Can I delete lines from a file without reading in my old file?

Answer 1

The general method to achieve this for logfiles is 'rotation' - when the logfiles gets older or hits a certain size, you rename it and start writing a new one. If you are using logging module, there is even a preconfigured one - RotatingFileHandler that does this automatically.

As for your question: you can only truncate from the back, not from the beginning. An approximate solution would be to seek() to 20% of the file, find first '\n' and copy it out - but it will be slow and prone to race conditions. Go with logging and RotatingFileHandler.

Answer 2

As others have said, the traditional way to solve this problem is to keep 5 different files instead of 1 large one. When you need to delete 20%, just delete the oldest file and rename the others.

As handy as text files are, you might also consider a database. It is designed to be able to delete any part of the data at any time.