Initializing and destroying Python multiprocessing workers

Question

I have a model which I'm calling many times from Python. The model takes a long time to startup and shutdown, but only a short time to process the input data (which can be done multiple times between startup/shutdown). The multiprocessing Pool() seemed like a good way to get this done, but I'm having trouble getting the Model() class to destory correctly.

A simplified structure of the programs code is given below. In reality, the init and del functions need to do some clever things with the win32com.client module, and the model.y variable is a handle to control an external application.

#!/usr/bin/env python

import multiprocessing
import random
import time

class Model():
    def __init__(self):
        self.y = random.randint(0,5) # simplification
        time.sleep(5) # initialisation takes a while...
    def __del__(self):
        del(self.y) # simplification
        time.sleep(1) # destruction isn't especially quick either

def init():
    global model
    model = Model()

def f(x): # worker function, runs quickly
    return model.y * x ** 2

def main():
    pool = multiprocessing.Pool(processes=2, initializer=init)
    it = pool.imap(f, range(4))
    for n in range(4):
        print it.next()
    pool.close()

if __name__ == '__main__':
    main()

The del function is never called for the Model()s, I'm guessing due to some reference being held in the garbage collector. How can I ensure that the model is closed correctly at the end of the program?

Answer 1

Solution of johnthexiii would kill the model at first run of worker function. You could offer a seperate destroy function:

import time

def g(x): # destroy function
    del model
    time.sleep(1) # to ensure, this worker does not pick too many
    return None

Before pool.close() you add

pool.map_async(g, range(2), 1) # if specified to have two processes before

I don’t think, this is a very “clean” solution, but it should work.