How to perform multiprocessing for a single function in Python?

Question

I am reading the Multiprocessing topic for Python 3 and trying to incorporate the method into my script, however I receive the following error:

AttributeError: __ exit __

I use Windows 7 with an i-7 8-core processor, I have a large shapefile which I want processed (with the mapping software, QGIS) using all 8 cores preferably. Below is the code I have, I would greatly appreciate any help with this matter:

from multiprocessing import Process, Pool

def f():
    general.runalg("qgis:dissolve", Input, False, 'LAYER_ID', Output)

if __name__ == '__main__':
    with Pool(processes=8) as pool:
        result = pool.apply_async(f)

Answer 1

The context manager feature of multiprocessing.Pool was only added into Python 3.3:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

The fact that __exit__ is not defined suggests you're using 3.2 or earlier. You'll need to manually call terminate on the Pool to get equivalent behavior:

if __name__ == '__main__':
    pool = Pool(processes=8)
    try:
        result = pool.apply_async(f)
    finally:
        pool.terminate()

That said, you probably don't want to use terminate (or the with statement, by extension) here. The __exit__ method of the Pool calls terminate, which will forcibly exit your workers, even if they're not done with their work. You probably want to actually wait for the worker to finish before you exit, which means you should call close() instead, and then use join to wait for all the workers to finish before exiting:

if __name__ == '__main__':
    pool = Pool(processes=8)
    result = pool.apply_async(f)
    pool.close()
    pool.join()