NGINX - Python - UWSGI kill issue

Question

After adding a runloop code in Python, uWSGI seems to be taking longer to kill.

---

Setup

• Python, Flask
• Running on Nginx with uWSGI
• Using a psql database

---

Issue

Stopping uWSGI used to be very quick. Recently I integrated a background thread to periodically check the database and make changes if needed, every 60 seconds.

This seems to be working just fine, except now every time I try to kill uWSGI, it takes a long time.

• 'seems' like the longer I leave the server running, the longer it takes to die,
• or maybe it always just gets killed after the current 60 second loop ends? (I'm not sure my visual inspection supports this)
• sounds like a leak?

---

Here is the code I recently added:

################################
## deploy.ini module .py file ##
################################
from controllers import runloop
from flask import Flask
from flask import request, redirect,Response
app = Flask(__name__)

runloop.startrunloop()

if __name__ == '__main__':
    app.run() #app.run(debug=True)

################################
## runloop.py ##
################################

### initialize run loop ###
## code ref: http://stackoverflow.com/a/22900255/2298002
# "Your additional threads must be initiated from the same app that is called by the WSGI server.
# 'The example below creates a background thread that executes every 5 seconds and manipulates data
#    structures that are also available to Flask routed functions."
#####################################################################

POOL_TIME = 60 #Seconds

# variables that are accessible from anywhere
commonDataStruct = {}
# lock to control access to variable
dataLock = threading.Lock()
# thread handler
yourThread = threading.Thread()

def startrunloop():
    logfuncname = 'runloop.startrunloop'
    logging.info(' >> %s >> ENTER ' % logfuncname)

    def interrupt():
        logging.info(' %s >>>> interrupt() ' % logfuncname)

        global yourThread
        yourThread.cancel()

    def loopfunc():
        logging.info(' %s >>> loopfunc() ' % logfuncname)

        global commonDataStruct
        global yourThread
        
        with dataLock:
            # Do your stuff with commonDataStruct Here
            
            # function that performs at most 15 db queries (right now)
            # this function will perform many times more db queries in production 
            auto_close_dws()
        
        # Set the next thread to happen
        yourThread = threading.Timer(POOL_TIME, loopfunc, ())
        yourThread.start()

    def initfunc():
        # Do initialisation stuff here
        logging.info(' %s >> initfunc() ' % logfuncname)

        global yourThread
        # Create your thread
        yourThread = threading.Timer(POOL_TIME, loopfunc, ())
        yourThread.start()

    # Initiate
    initfunc()
    # When you kill Flask (SIGTERM), clear the trigger for the next thread
    atexit.register(interrupt)

---

Additional info (all flask requests work just fine):

I start server with:

$ nginx

and stop with:

$ nginx -s stop

I start uWSGI with:

$ uwsgi —enable-threads —ini deploy.ini

I stop uWSGI to make python changes with:

ctrl + c (if in the foreground)

Otherwise I stop uWSGI with:

$ killall -s INT uwsgi

Then after making changes to the Python code, I start uWSGI again with:

$ uwsgi —enable-threads —ini deploy.ini

The following is an example Nginx output when I try to kill:

^CSIGINT/SIGQUIT received...killing workers...
Fri May  6 00:50:39 2016 - worker 1 (pid: 49552) is taking too much time to die...NO MERCY !!!
Fri May  6 00:50:39 2016 - worker 2 (pid: 49553) is taking too much time to die...NO MERCY !!!

---

Any help or hints are greatly appreciated. Please let me know if I need to be more clear with anything or if I’m missing any details.

Answer 1

I know the question is a bit old, but I had the same problem and Google got me here, so I will answer for anyone who gets here in the same boat.

The problem seems to be caused by the --enable-threads option, we have several applications running with uwsgi and flask and only the one with this option has the problem.

If what you want is to have the uwsgi process dying faster, you can add this options:

reload-mercy = *int*
worker-reload-mercy = *int*

They will cause the uwsgi to force the process to quit after int seconds.

On the other hand, if all you need is to reload the uwsgi, try just sending a SIGHUP signal. This will cause the uwsgi to reload its children.

POST NOTE: It seems I had spoken too soon, using SIGHUP also hangs sometimes. I am using the mercy options to avoid the hanging to take too long.

Also, I found the issue report on uwsgi github, if anyone wants to follow it:

https://github.com/unbit/uwsgi/issues/844