Async multiprocessing python

Question

So I've read this nice article about asynch threads in python. Tough, the last one have some troubles with the GIL and threads are not as effective as it may seems.

Luckily python incorporates Multiprocessing which are designed to be not affected by this trouble.

I'd like to understand how to implement a multiprocessing queue (with Pipe open for each process) in an async manner so it wouldn't hang a running async webserver .

I've read this topic however I'm not looking for performance but rather boxing out a big calculation that hangs my webserver. Those calculations require pictures so they might have a significant i/o exchange but in my understanding this is something that is pretty well handled by async.

All the calcs are separate from each other so they are not meant to be mixed.

I'm trying to build this in front of a ws handler.

If you hint heresy in this please let me know as well :)

Answer 1

This is re-sourced from a article after someone nice on #python irc hinted me on async executors, and another answer on reddit :

(2) Using ProcessPoolExecutor “The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.”

import asyncio
from concurrent.futures import ProcessPoolExecutor

def cpu_heavy(num):
    print('entering cpu_heavy', num)
    import time
    time.sleep(10)
    print('leaving cpu_heavy', num)
    return num

async def main(loop):
    print('entering main')
    executor = ProcessPoolExecutor(max_workers=3)
    data = await asyncio.gather(*(loop.run_in_executor(executor, cpu_heavy, num) 
                                  for num in range(3)))
    print('got result', data)
    print('leaving main')


loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

And this from another nice guy on reddit ;)