os.kill vs process.terminate within aiohttp

Question

I have this little piece of code

import asyncio
from aiohttp import web
from multiprocessing import Process
import time

def block():
    i = 0
    while i < 5:
        print('running')
        time.sleep(1)
        i += 1


async def start(request):
    process = Process(target=block)
    process.start()
    print('process started')
    request.app['p'] = process
    return web.Response(status=200)

async def stop(request):
    if request.app['p'].is_alive():
        print('stop process')
        request.app['p'].terminate()
        return web.Response(status=200)

app = web.Application()
app.router.add_route('GET', '/start', start)
app.router.add_route('GET', '/stop', stop)
web.run_app(app, host='127.0.0.1', port=8888)

When I send an HTTP GET on /start, I can see running being output on the terminal each second like expected. But when I send an HTTP GET on /stop, the stream of running doesn't stop until the loop is finished. Furthermore, when the loop finishes, the whole application dies.

here's an output of this:

❯ python3 server.py
======== Running on http://127.0.0.1:8888 ========
(Press CTRL+C to quit)
process started
running
running
running
stop process
running
running

❯

Interestingly this request handler does work without stopping the whole application.

async def stop2(request):
    if request.app['p'].is_alive():
        print('stop2 process')
        os.kill(request.app['p'].pid, signal.SIGKILL)
        return web.Response(status=200)

So what is the difference between using os.kill and process.terminate with python/aiohttp?

Answer 1

The difference is that terminate() sends SIGTERM signal, and kill() sends the signal that you specify.

By default, Aiohttp registers handlers for SIGTERM and SIGINT on startup for graceful shutdown. In production environments, you typically don't run your application in console (so you cannot disable it with Ctrl+C), but with some process supervisor, like Systemd or Docker. To stop such application, the supervisor typically sends SIGTERM. You can disable default handlers with arg handle_signals=False to run_app() call, but when you decide to shutdown the server, any of these signals would be an instant kill.

As you use multiprocessing module on UNIX, your child process is made with fork() call - that means, when created, it is almost complete copy of main process - and it has a copy of parent signal handlers. Using subprocess module helps with that - you will have brand new python process made from scratch, without parent handlers. Or you can just restore default handlers, if you add these lines to the beginning of block() call:

signal.set_wakeup_fd(-1)
signal.signal(signal.SIGTERM, signal.SIG_DFL)

Typically, signal is consumed by only one process (you specify its pid in kill()). But with aiohttp some magic happens. Underlying implementation of asyncio uses signal.set_wakeup_fd() function to handle signals, which makes signals be written to a socket. This makes signal just a standard event for asyncio event loop. But this is where your application breaks. Not only your worker process is not stopped (because aiohttp interrupted the signal). As the socket was created before fork() call, it is shared between your main process and all children processes. So any SIGTERM and SIGINT signals to your child processes will be handled by main process. Your webserver is already dead after /stop request - even before the loop finishes.