Python multiprocessing - TypeError: 'Nonetype' object not callable

Question

I have encountered an issue with the Python multiprocessing library when using PySQLPool. It results in me getting the following exception: 'TypeError: 'NoneType' object is not callable'.

I create a Pool using multiprocessing, and call Pool.map on a dummy function. In the arguments to map I include a connection from PySQLPool. I have included the minimum code I could reproduce the bug with below:

from config import *
import PySQLPool
import multiprocessing as mp
def a(b):
    return b
PySQLPool.getNewPool().maxActiveConnections = 5
connection = PySQLPool.getNewConnection(user=USER,
                    passwd=PASSWORD,
                    host='localhost',
                    db=DATABASE,
                    use_unicode=True,
                    charset='utf8',
                    commitOnEnd=False)
pool = mp.Pool(processes=8)
args = [(connection)]
result_list = pool.map(a, args, 8)
pool.close()
pool.join()

It results in the following error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/root/.pyenv/versions/2.7.6/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/root/.pyenv/versions/2.7.6/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/root/.pyenv/versions/2.7.6/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
    put(task)
TypeError: 'NoneType' object is not callable

I have the following installed with pip

(venv)root@Ubuntu-1204-precise-64-minimal:/usr/share/python-multiproc/src# pip list
MySQL-python (1.2.5)
pip (6.0.8)
PySQLPool (0.3.8)
setuptools (12.0.5)

I have tested it on Ubuntu 12.04 with Python version 2.7.3, 2.7.6, and 2.7.9. Also tested on Windows 8 with Python 2.7.9. Furthermore I have tried with both PySQLPool 0.3.8 and 0.4.

What is the cause of this and how can I debug it?

Answer 1

This obscure error can come from corrupted arguments passed to the called function. This can happen with complex objects due to pickling/unpickling logic.

If possible try reworking your code to pass the absolute minimum number of args to your function. This may mean to re-create some objects inside each process.