Asynchronously receive output from long running shell commands with asyncio (Python 3.4+)?

Question

I'm trying to figure out how to simply start a number of long running shell commands in a non-blocking way, and asynchronously handle their output when they finish, in the order they finish, even if that is another order than they started, using the asyncio python library available in Python 3.4 and forward.

I couldn't find a simple example of doing this, even in the asyncio documentation itself, which also seems to be quite low-level.

Answer 1

Use get_lines() coroutines, to get shell commands output asynchronously and pass the coroutines to asyncio.as_completed(), to get the results in the order they finish:

#!/usr/bin/env python3.5
import asyncio
import sys
from asyncio.subprocess import PIPE, STDOUT

async def get_lines(shell_command):
    p = await asyncio.create_subprocess_shell(shell_command,
            stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    return (await p.communicate())[0].splitlines()

async def main():
    # get commands output concurrently
    coros = [get_lines('"{e}" -c "print({i:d}); import time; time.sleep({i:d})"'
                       .format(i=i, e=sys.executable))
             for i in reversed(range(5))]
    for f in asyncio.as_completed(coros): # print in the order they finish
        print(await f)


if sys.platform.startswith('win'):
    loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()