How to do virtual file processing?

Question

So for creating files I use the following:

fileHandle = open('fileName', 'w') 

then write the contents to the file, close the file. In the next step I process the file. At the end of the program, I end up with a "physical file" that I need to delete.

Is there a way to write a "virtual" file that behaves exactly like a "physical" one (allowing it to be manipulated the same way) but does not exist at the end of the run in Python?

Answer 1

You might want to consider using a tempfile.SpooledTemporaryFile which gives you the best of both worlds in the sense that it will create a temporary memory-based virtual file initially but will automatically switch to a physical disk-based file if the data held in memory exceeds a specified size.

Another nice feature is that (when using memory) it will automatically use either an io.BytesIO or io.StringIO depending on what mode is being used—allowing you to either read and write Unicode strings or binary data (bytes) to it.

The only tricky part might be the fact that you'll need to avoid closing the file between steps because doing so would cause it to be deleted from memory or disk. Instead you can just rewind it back to the beginning with a file seek(0) method call.

When you are completely done with the file and close it, it will automatically be deleted from disk if the amount of data in it caused it to be rolled-over to a physical file.