Preventing file handle inheritance in multiprocessing lib

Question

Using multiprocessing on windows it appears that any open file handles are inherited by spawned processes. This has the unpleasant side effect of locking them.

I'm interested in either:
1) Preventing the inheritance
2) A way to release the file from the spawned process

Consider the following code which works fine on OSX, but crashes on windows at os.rename

from multiprocessing import Process
import os

kFileA = "a.txt"
kFileB = "b.txt"

def emptyProcess():
    while 1:
        pass

def main():
    # Open a file and write a message
    testFile = open(kFileA, 'a')
    testFile.write("Message One\n")

    # Spawn a process
    p = Process(target=emptyProcess)
    p.start()

    # Close the file
    testFile.close()

    # This will crash
    # WindowsError: [Error 32] The process cannot access the file
    #               because it is being used by another process
    os.rename(kFileA, kFileB)

    testFile = open(kFileA, 'a')
    testFile.write("Message Two\n")
    testFile.close()

    p.terminate()


if __name__ == "__main__":
    main()

Answer 1

The fileno() method returns the file number as assigned by the runtime library. Given the file number, you can then call msvcrt.get_osfhandle() to get the Win32 file handle. Use this handle in the call to SetHandleInformation. So something like the following may work:

win32api.SetHandleInformation(
    msvcrt.get_osfhandle(testFile.fileno()),
    win32api.HANDLE_FLAG_INHERIT,
    0)

I'm not certain of the exact usage of the win32api module, but this should help bridge the gap between a Python file object and a Win32 handle.

Answer 2

I don't know about the multiprocessing module, but with the subprocess module you can instruct it to not inherit any file descriptors:

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. (Unix only). Or, on Windows, if close_fds is true then no handles will be inherited by the child process. Note that on Windows, you cannot set close_fds to true and also redirect the standard handles by setting stdin, stdout or stderr.

Alternatively you could close all file descriptors in your child process with os.closerange

Close all file descriptors from fd_low (inclusive) to fd_high (exclusive), ignoring errors. Availability: Unix, Windows.