Why can't this Python multiprocessing process be terminated from Kivy?

Question

I'm trying to run a django development server from within a Kivy application. This did work out quite well so far.

Now i want to allow the user to continue working with the program while the server is running. My idea was to create a multiprocessing.Process for the httpd.serve_forever() to avoid a complete lock of the main program. Did work well. This is the code in my internal_django module:

import multiprocessing
import os
import time

from wsgiref.simple_server import make_server

def django_wsgi_application():

    PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
    settings_module = "djangosettings"#%s.djangosettings" % PROJECT_ROOT.split(os.sep)[-1]

    os.environ.update({"DJANGO_SETTINGS_MODULE":settings_module})

    from django.core.wsgi import get_wsgi_application
    application = get_wsgi_application()

    return application


class Singleton(type):
    _instances = {}
    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class DjangoServer():
    __metaclass__ = Singleton

    def start(self):
        self.httpd = make_server('', 8000, django_wsgi_application())
        self.server = multiprocessing.Process(target=self.httpd.serve_forever)
        self.server.start()
        print "Now serving on port 8000..."
        print "Server Process PID = %s" %self.server.pid

    def stop(self):
        print("shutdown initiated")
        print "Server Process PID = %s" %self.server.pid
        while self.server.is_alive():
            self.server.terminate()
            print("Server should have shut down")
            time.sleep(1)
        print("Server is_alive: %s" %self.server.is_alive())
        self.server.join()
        print("server process joined")



if __name__ == "__main__":
    server = DjangoServer()
    server.start()
    time.sleep(3)
    server.stop()

When i run this code, everything works as expected. This is what is being put out in the console:

Now serving on port 8000...
Server Process PID = 1406
shutdown initiated
Server Process PID = 1406
Server should have shut down
Server is_alive: False
server process joined

Next step was to provide a way to stop the server from within the Kivy application. For that i just wanted to use my DjangoServer class as i did before:

from internal_django import DjangoServer

class StartScreen(Screen): 
    def start_server(self):
        server = DjangoServer()
        server.start()


class StopScreen(Screen):  
    def stop_server(self):
        server = DjangoServer()
        server.stop()

But when doing so, the process once started never quits. My first idea was that the Singleton did not work as expected, and that i try to quit the wrong process. but as you can see in the output, the PID's are identical. The server receives the terminate command, but just continues to work. This is what the console looks like:

Now serving on port 8000...
Server Process PID = 1406
shutdown initiated
Server Process PID = 1406
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down
Server should have shut down

(and so on, until i manually kill the server process)

Am i using multiprocessing in a completely wrong way? Is Kivy somehow interfering with the process?

Answer 1

I think the problems here might be two:

1. A signal handler is intercepting the TERM request sent by Process.terminate() and ignores it. To verify that simply use the signal.getsignal(signal.SIGTERM) from within the new process and print the results. To circumvent such issue you can reset the default behavior with signal.signal(signal.SIGTERM, signal.SIG_DFL), nevertheless keep in mind that there might be a reason why SIGTERM is silenced by the frameworks (I'm not familiar neither with Django nor with Kivy).

2. If you're using Python 2 you must consider that the interpreter does not process signals if it's blocked on a synchronization primitive from threading library (Locks, Semaphores..) or on a native C call. The serve_forever() function might fall in these cases (use the force of the source!). Quick check could be trying to run the code on Python 3 and see whether it works or not.

A quick and dirty solution consists in waiting a small amount of time and send a SIGKILL if the process is still alive.

import os
import signal

process.terminate()
process.join(1)

if process.is_alive() and os.name != 'nt':
    try:
        os.kill(process.pid, signal.SIGKILL)
        process.join()
    except OSError:
        return  # process might have died while checking it

On windows you cannot kill a process in such simple way that's why I test the os.name.

It's a pretty raw approach so I'd rather recommend to find the cause of the issue.

Answer 2

What happens if you call terminate(), then join() and skip the while loop? Also, I shuffle the code a little and factor some code into _create_server(). Please let me know if this works out for you.

class DjangoServer():
    __metaclass__ = Singleton

    def _create_server(self):
        httpd = make_server('', 8000, django_wsgi_application())
        print "Now serving on port {}...".format(httpd.server_port)
        httpd.serve_forever()

    def start(self):
        self.server = multiprocessing.Process(target=self._create_server)
        self.server.start()
        print "Server Process PID = %s" %self.server.pid

    def stop(self):
        print("shutdown initiated")
        print "Server Process PID = %s" %self.server.pid
        self.server.terminate()
        self.server.join()
        print("server process terminated")