'Collection' object is not callable error in pymongo with process Pool

Question

Using the code below causes:

'Collection' object is not callable. If you meant to call the '__getnewargs__' method on a 'Collection' object it is failing because no such method exists.

The code : from multiprocessing import Pool db = MongoClient(ip,port)

def f(cursor, arg):
    for doc in cursor:
       ...

p = Pool(4)
for arg in args:
    cursor = db[dbName][collName].find()
    p.apply_async(f,[cursor, arg])

db.close()

Can't figure out what is the problem and how to debug the code.

Full Traceback:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 808, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\multiprocessing\pool.py", line 342, in _handle_tasks
    put(task)
  File "C:\Python27\lib\site-packages\pymongo\collection.py", line 1489, in __call__
    self.__name.split(".")[-1])
TypeError: 'Collection' object is not callable. If you meant to call the '__getnewargs__' method on a 'Collection' object it is failing because no such method exists.

Answer 1

You have a problem in your use of cursor. The Collection.find method returns a Cursor object which is a consumable. (http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.getitem) I don't know if this is the cause of the exception but it is surely a problem.

Two solutions for you:

1. Explicitly pull the documents before threading using a [:] or list
2. Give the cursor in apply_async and clone the cursor using clone method (http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.clone)