Multiprocessing : NULL result without error in PyObject_Call

Question

Here is a sample program where I use multiprocessing. The calculations are done with multiprocessing.Process and the results are collected using multiprocessing.Queue.

#THIS PROGRAM RUNS WITH ~40Gb RAM. (you can reduce a,b,c for less RAM 
#but then it works for smaller values)
#PROBLEM OCCURS ONLY FOR HUGE DATA.   
from numpy import *
import multiprocessing as mp

a = arange(0, 3500, 5)
b = arange(0, 3500, 5)
c = arange(0, 3500, 5)  
a0 = 540. #random values
b0 = 26.
c0 = 826.
def rand_function(a, b, c, a0, b0, c0):
    Nloop = 100.
    def loop(Nloop, out):
        res_total = zeros((700, 700, 700), dtype = 'float') 
        n = 1
        while n <= Nloop:
            rad = sqrt((a-a0)**2 + (b-b0)**2 + (c-c0)**2)
            res_total += rad
            n +=1 
        out.put(res_total)
    out = mp.Queue() 
    jobs = []
    Nprocs = mp.cpu_count()
    print "No. of processors : ", Nprocs
    for i in range(Nprocs):
        p = mp.Process(target = loop, args=(Nloop/Nprocs, out)) 
        jobs.append(p)
        p.start()

    final_result = zeros((700, 700, 700), dtype = 'float')

    for i in range(Nprocs):
        final_result = final_result + out.get()

    p.join()
test = rand_function(a,b,c,a0, b0, c0)

Here is the error message :

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
    send(obj)
SystemError: NULL result without error in PyObject_Call

I read here that it is a bug. But I am unable to understand. Can anyone please tell me any way out to calculate huge data using multiprocessing?

Thank you very much

Answer 1

The bug report your reference states that multiprocessing module is unable to push huge arguments to subprocess.

The reason is that it needs to pickle these arguments and store the pickled blob somewhere in memory.

You, however, don't need to pass arrays as arguments.

Possible causes:

• passing a closure loop as a target
• passing mp.Queue() as argument

Please see http://stevenengelhardt.com/2013/01/16/python-multiprocessing-module-and-closures/ about converting your closure to a class.

Set up full state before you give control to multiprocessing.