Python: capturing all writes to a file in memory

Question

Is there some way to "capture" all attempted writes to a particular file /my/special/file, and instead write that to a BytesIO or StringIO object instead, or some other way to get that output without actually writing to disk?

The use case is: there's a 'handler' function, whose contract is that it should write its output to /my/special/file. I don't have any control over this handler function -- I don't write it, I don't know its contents and I can't change its contents, and the contract cannot change. I'd like to be able to do something like this:

# 'output' has whatever 'handler' has written to `/my/special/file`
output = handler.run(data) 

Even if this is an odd request, I'd like to be able to do this even with a 'hackier' answer.

EDIT: my code (and handler) will be invoked many times on a lot of chunks of data, so performance (both latency and throughput) are important.

Thanks.

Answer 1

If you're talking about code in your own Python program, you could monkey-patch the built in open function before that code gets called. Here's a really stupid example, but it shows that you can do this. This causes code that thinks it's writing to a file to instead write into an in-memory buffer. The calling code then prints what the foreign code wrote to the file:

import io

# The function you don't have access to that writes to a file
def foo():
    f = open("/tmp/foo", "w")
    f.write("blahblahblah\n")
    f.close()

# The buffer to contain the captured text
capture_buffer = ""

# My silly file-like object that only handles write(str) and close()
class MyFileClass:
    def write(self, str):
        global capture_buffer
        capture_buffer += str
    def close(self):
        pass

# patch open to return a MyFileClass instance
def my_open2(*args, **kwargs):
    return MyFileClass()
open = my_open2


# Call the target function
foo()

# Print what the function wrote to "the file"
print(capture_buffer)

Result:

blahblahblah

Sorry for not spending more time with this. Just showing you it's possible. As others say, a mocking module might be the way to go to not have to grow your own thing here. I don't know if they allow access to what is written. I guess they must. Such a module is just going to do a better job of what I've shown here.

If your program does other file IO with open, or whichever method the mystery code uses to open the file, you'd check the incoming path and only return your special object if it was the one path you're interested in. Otherwise, you could just call the original open, which you could stash away under another name.