prevent __del__ from being called in multiprocessing

Question

When playing around with multiprocessing I noticed that in the following script, __del__ is called twice (once in the child processes and once in the parent).

class myclass(object):

    def __init__(self,val):
        self.val=val
        print ("Initializing %s"%str(self.val))

    def __del__(self):
        print ("deleting %s"%str(self.val))


if __name__ == "__main__":
    import multiprocessing
    p=multiprocessing.Pool(4)
    obj_list=p.map(myclass,range(30))
    raw_input()

For this script, it doesn't matter ... but what if __del__ has side-effects? (One possible example that I can think of would be to release some sort of lock file). Is there a way to prevent __del__ from being called twice?

Answer 1

__del__ is normally used to finalize an object, not an external resource, so it makes sense to call it in both processes (since, of course, both have their own copies of the object after a fork). It's not a good idea to try to prevent any use of __del__ in a given process, but in places where you really, really need it to close out an external resource like a file on the filesystem, which doesn't get duplicated by a fork, your destructor should just check to see if the expected resource really needs cleaning up before doing so. That may or may not be the case for "releasing some kind of lock file", depending on how you're implementing that.

If it's an option, you might want to look at doing resource acquisition and release using "with" context managers instead of depending on the vagaries of garbage collection.

Answer 2

Something like releasing a lock shouldn't be something to do in a __del__ method, it's much better to use the lock as context manager in a with statement to ensure the release of the lock.

And besides that, the generated instances of the classes live in different processes, changing anything in a subprocess won't affect the other subrpcesses, of course with exception of the objects used for IPC (like queues, pipes, locks, ... from the multiprocessing module).

Another thing is this: the instances returned to the main process are not the same instances created in the subprocesses. Return values in multiprocessing are pickled in the child processes, sent to the parent and unpickled there. This process involves a ForkingPickler*. therfore __del__ isn't called multiple times on the same instances, but on different instances in different subprocesses.

*: not completely sure what exactly is going on here, maybe somebody else knows more... but maybe a different version of the example will help:

import multiprocessing
import time
import os

class myclass(object):

    def __init__(self,val):
        self.val=val
        print ("Initializing %s - %s - %s" % (str(self.val), self, os.getpid()))

    def __del__(self):
        print ("Deleting     %s - %s - %s" % (str(self.val), self, os.getpid()))

if __name__ == "__main__":
    p = multiprocessing.Pool(3)
    obj_list=p.map(myclass,range(5))
    del p
    for o in obj_list:
        print ("Returned     %s - %s - %s" % (str(o.val), o, os.getpid()))

output:

Initializing 0 - <__main__.myclass object at 0x7f2497fdc0d0> - 7574
Initializing 2 - <__main__.myclass object at 0x7f2497fdc110> - 7574
Deleting     0 - <__main__.myclass object at 0x7f2497fdc0d0> - 7574
Initializing 1 - <__main__.myclass object at 0x7f2497fdc150> - 7575
Initializing 3 - <__main__.myclass object at 0x7f2497fdc1d0> - 7575
Deleting     1 - <__main__.myclass object at 0x7f2497fdc150> - 7575
Initializing 4 - <__main__.myclass object at 0x7f2497fdc0d0> - 7574
Deleting     2 - <__main__.myclass object at 0x7f2497fdc110> - 7574
Returned     0 - <__main__.myclass object at 0x7f2497fdc650> - 7573
Returned     1 - <__main__.myclass object at 0x7f2497fdc7d0> - 7573
Returned     2 - <__main__.myclass object at 0x7f2497fdc810> - 7573
Returned     3 - <__main__.myclass object at 0x7f2497fdc850> - 7573
Returned     4 - <__main__.myclass object at 0x7f2497fdc890> - 7573
Deleting     3 - <__main__.myclass object at 0x7f2497fdc1d0> - 7575
Deleting     4 - <__main__.myclass object at 0x7f2497fdc890> - 7573
Deleting     3 - <__main__.myclass object at 0x7f2497fdc850> - 7573
Deleting     2 - <__main__.myclass object at 0x7f2497fdc810> - 7573
Deleting     1 - <__main__.myclass object at 0x7f2497fdc7d0> - 7573
Deleting     0 - <__main__.myclass object at 0x7f2497fdc650> - 7573

note the different process and object ids