asynchronous subprocess using flask-aiohttp

Question

I'm using a Flask web server to provide an interface to a time-consuming calculation. To increase performance, I want to

1. start the calculation as a new subprocess to be able to use multiple CPU cores for multiple concurrent calculations
2. let the calculations run asynchronously using asyncio

To call asyncio coroutines from Flask I started using flask-aiohttp, which is working nicely for simple delay tasks as shown in the examples. However, I fail to call the asynchronous subprocess from inside Flask:

#!/usr/bin/env python3
# coding: utf-8

from flask import Flask
from flask.ext.aiohttp import AioHTTP, async

import asyncio
from asyncio.subprocess import PIPE

CALC_SCRIPT = './calc'

app = Flask(__name__)
aio = AioHTTP(app)

@app.route('/calc/<int:n>')
@async
def calc(n):
    print('calc({}) called'.format(n))
    create = asyncio.create_subprocess_exec(CALC_SCRIPT, str(n),
                                            stdout=PIPE, stderr=PIPE)
    print('create... ', end='')
    process = yield from create
    print('process created. {!r}, type={}'.format(process,
                                                  type(process)))
    yield from process.wait()
    print('process finished.')

    # yields (stdout, stderr)
    result = '\n'.join(ch.decode().rstrip() for ch in
                        (yield from process.communicate()) if ch)
    return result

if __name__ == '__main__':
    aio.run(app, debug=True)

The process is being created, but never returns:

GET http://127.0.0.1:5000/calc/5
calc(5) called
creating... process created. <Process 5647>,
    type=<class 'asyncio.subprocess.Process'>

What am I doing wrong?

Answer 1

Reason: running asyncio subprocesses from sub-threads has limits, see asyncio docs Subprocess and threads.

Detail: with debug=True, Flask-aiohttp handle requests in a sub-thread started by Werkzeug's run_with_reloader. Turn debug off and your code runs correctly.

Alternatively according to the docs above, Flask-aiohttp should add a call to asyncio.get_child_watcher() right before calling run_with_reloader. With this call, your code runs even with debug=True.