Open a file in memory

Question

(I'm working on a Python 3.4 project.)

There's a way to open a (sqlite3) database in memory :

    with sqlite3.connect(":memory:") as database:

Does such a trick exist for the open() function ? Something like :

   with open(":file_in_memory:") as myfile:

The idea is to speed up some test functions opening/reading/writing some short files on disk; is there a way to be sure that these operations occur in memory ?

Answer 1

How about StringIO:

import StringIO

output = StringIO.StringIO()
output.write('First line.\n')
print >>output, 'Second line.'

# Retrieve file contents -- this will be
# 'First line.\nSecond line.\n'
contents = output.getvalue()

# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()

python3: io.StringIO

Answer 2

There is something similar for file-like input/output to or from a string in io.StringIO.

There is no clean way to add url-based processing to normal file open, but being Python dynamic you could monkey-patch standard file open procedure to handle this case.

For example:

from io import StringIO

old_open = open
in_memory_files = {}

def open(name, mode="r", *args, **kwargs):
     if name[:1] == ":" and name[-1:] == ":":
          # in-memory file
          if "w" in mode:
               in_memory_files[name] = ""
          f = StringIO(in_memory_files[name])
          oldclose = f.close
          def newclose():
              in_memory_files[name] = f.getvalue()
              oldclose()
          f.close = newclose
          return f
     else:
          return old_open(name, mode, *args, **kwargs)

after that you can write

f = open(":test:", "w")
f.write("This is a test\n")
f.close()

f = open(":test:")
print(f.read())

Note that this example is very minimal and doesn't handle all real file modes (e.g. append mode, or raising the proper exception on opening in read mode an in-memory file that doesn't exist) but it may work for simple cases.

Note also that all in-memory files will remain in memory forever (unless you also patch unlink).

PS: I'm not saying that monkey-patching standard open or StringIO instances is a good idea, just that you can :-D

PS2: This kind of problem is solved better at OS level by creating an in-ram disk. With that you can even call external programs redirecting their output or input from those files and you also get all the full support including concurrent access, directory listings and so on.

Answer 3

io.StringIO provides a memory file implementation you can use to simulate a real file. Example from documentation:

import io

output = io.StringIO()
output.write('First line.\n')
print('Second line.', file=output)

# Retrieve file contents -- this will be
# 'First line.\nSecond line.\n'
contents = output.getvalue()

# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()

In Python 2, this class is available instead as StringIO.StringIO.