How to pass file descriptors from parent to child in python?

Question

I am using multiprocessing module, and using pools to start multiple workers. But the file descriptors which are opened at the parent process are closed in the worker processes. I want them to be open..! Is there any way to pass file descriptors to be shared across parent and children?

Answer 1

On Python 2 and Python 3, functions for sending and receiving file descriptors exist in multiprocessing.reduction module.

Example code (Python 2 and Python 3):

import multiprocessing
import os

# Before fork
child_pipe, parent_pipe = multiprocessing.Pipe(duplex=True)

child_pid = os.fork()

if child_pid:
    # Inside parent process
    import multiprocessing.reduction
    import socket
    # has socket_to_pass socket object which want to pass to the child
    socket_to_pass = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    socket_to_pass.connect("/dev/log")
    # child_pid argument to send_handle() can be arbitrary on Unix,
    # on Windows it has to be child PID
    multiprocessing.reduction.send_handle(parent_pipe, socket_to_pass.fileno(), child_pid)
    socket_to_pass.send("hello from the parent process\n".encode())
else:
    # Inside child process
    import multiprocessing.reduction
    import socket
    import os
    fd = multiprocessing.reduction.recv_handle(child_pipe)
    # rebuild the socket object from fd
    received_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
    # socket.fromfd() duplicates fd, so we can close the received one
    os.close(fd)
    # and now you can communicate using the received socket
    received_socket.send("hello from the child process\n".encode())

Answer 2

There is also a fork of multiprocessing called multiprocess, which replaces pickle with dill. dill can pickle file descriptors, and thus multiprocess can easily pass them between processes.

>>> f = open('test.txt', 'w')
>>> _ = map(f.write, 'hello world')
>>> f.close()
>>> import multiprocess as mp
>>> p = mp.Pool()
>>> f = open('test.txt', 'r')
>>> p.apply(lambda x:x, f)
'hello world'
>>> f.read()
'hello world'
>>> f.close()

Answer 3

multiprocessing itself has helper methods for transferring file descriptors between processes on Windows and Unix platforms that support sending file descriptors over Unix domain sockets in multiprocessing.reduction: send_handle and recv_handle. These are not documented but are in the module's __all__ so it may be safe to assume they are part of the public API. From the source it looks like these have been available since at least 2.6+ and 3.3+.

All platforms have the same interface:

• send_handle(conn, handle, destination_pid)
• recv_handle(conn)

Where:

• conn (multiprocessing.Connection): connection over which to send the file descriptor
• handle (int): integer referring to file descriptor/handle
• destination_pid (int): integer pid of the process that is receiving the file descriptor - this is currently only used on Windows