Why I cannot use python module concurrent.futures in class method?

Question

I want to make my class method runs parallel, but it only produces some kind of error that I can not solve. My code is:

import concurrent.futures as futures

samples = ['asfd', 'zxcv', 'asf', 'qwer']

class test:
    def __init__(self, samples):
        maturedb = {}
        with futures.ProcessPoolExecutor() as exe:
            for samplename, dResult in exe.map(self.make_readdb, samples):
                maturedb[samplename] = dResult
        print(maturedb)

    def make_readdb(self, samplename):
        return samplename, 1

test(samples)

If I run this code in Ubuntu machine, an Error like below occurs:

Traceback (most recent call last):
    File "/usr/lib/python3.2/multiprocessing/queues.py", line 272, in _feedsend(obj)
    _pickle.PicklingError: Can't pickle <class 'method'>: attribute lookup builtins.method failed

The method make_readdb is just simplified to make an example, but it is an bottleneck in real code and I need to make it parallel.

Answer 1

From the docs:

The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.

Try a ThreadPoolExecutor

I looked over your code again, the problem is that the function - make_readdb - is a member of the class test. Can you refactor and pull this function out?