uwsgi master graceful shutdown

Question

I'm running uwsgi+flask application, The app is running as a k8s pod.

When i deploy a new pod (a new version), the existing pod get SIGTERM.

This causes the master to stop accepting new connection at the same moment, what causes issues as the LB still pass requests to the pod (for a few more seconds).

I would like the master to wait 30 sec BEFORE stop accepting new connections (When getting SIGTERM) but couldn't find a way, is it possible?

My uwsgi.ini file: [uwsgi]

;https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html
http = :8080
wsgi-file = main.py
callable = wsgi_application
processes = 2
enable-threads = true
master = true
reload-mercy = 30
worker-reload-mercy = 30
log-5xx = true
log-4xx = true
disable-logging = true
stats = 127.0.0.1:1717
stats-http = true
single-interpreter= true
;https://github.com/containous/traefik/issues/615
http-keepalive=true
add-header = Connection: Keep-Alive

Answer 1

Seems like this is not possible to achieve using uwsgi:

https://github.com/unbit/uwsgi/issues/1974

The solution - (as mentioned on this kubernetes issue):

https://github.com/kubernetes/contrib/issues/1140

Is to use the prestop hook, quite ugly but will help to achieve zero downtime:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep","5"]

The template is taken from this answer: https://stackoverflow.com/a/39493421/3659858

Answer 2

Another option is to use the CLI option:

--hook-master-start "unix_signal:15 gracefully_kill_them_all"

or in the .ini file:

hook-master-start = "unix_signal:15 gracefully_kill_them_all"

which will gracefully terminate workers after receiving a SIGTERM (signal 15).

See the following for reference.

When I tried the above though, it didn't work as expected from within a docker container. Instead, you can also use uWSGI's Master FIFO file. The Master FIFO file can be specified like:

--master-fifo <filename>

or

master-fifo = /tmp/master-fifo

Then you can simply write a q character to the file and it will gracefully shut down your workers before exiting.