Is it possible to store python objects (specifically sklearn models) in memory mapped files?

Question

I have several large objects (sklearn models) that take up a lot of memory, and I want to share them between several process. is there a way to do this?

• It has to be the "live" object, and not a serialized version
• I know that there's a memory mapped version of numpy array, which are responsible for a significant part of the model memory - but using them will require significant changes to the sklearn source code, which would be hard to maintain

Answer 1

Under the proviso that the processes are launched from the same python script, here is an example that creates a second process and shares variables between the two processes. It is straightforward to elaborate on this to create some number of processes. Notice the constructs used to create and access the shared variables and lock. I have inserted a loop over an arithmetic process to generate some cpu usage so that you can monitor and see how this runs on a multi-core or multi-processor platform. Also note the use of a shared variable to control the second process, in this instance to tell it when to exit. And finally, the shared object can be a value or an array, see https://docs.python.org/2/library/multiprocessing.html

#!/usr/bin/python

from time import sleep
from multiprocessing import Process, Value, Lock

def myfunc(counter, lock, run):

    while run.value:
        sleep(1)
        n=0
        for i in range(10000):
            n = n+i*i
        print( n )
        with lock:
            counter.value += 1
            print( "thread %d"%counter.value )

    with lock:
        counter.value = -1
        print( "thread exit %d"%counter.value )

# =======================

counter = Value('i', 0)
run = Value('b', True)
lock = Lock()

p = Process(target=myfunc, args=(counter, lock, run))
p.start()

while counter.value < 5:
    print( "main %d"%counter.value )
    n=0
    for i in range(10000):
        n = n+i*i
    print( n )
    sleep(1)

with lock:
    counter.value = 0

while counter.value < 5:
    print( "main %d"%counter.value )
    sleep(1)

run.value = False

p.join()

print( "main exit %d"%counter.value)