uWSGI workers stuck: why

Question

I'm using uwsgi version 2.0.13.1 with the following config:

bin/uwsgi  -M -p 5 -C -A 4 -m -b 8192 -s :3031 --wsgi-file bin/django.wsgi --pidfile var/run/uwsgi.pid --touch-reload=var/run/reload-uwsgi.touch --max-requests=1000 --reload-on-rss=450 --py-tracebacker var/run/pytrace --auto-procname --stats 127.0.0.1:3040 --threads 40 --reload-mercy 600 --listen 200

(absolute path names cut)

When I run uwsgitop, all 5 workers appear as busy. When I try to get the stack trace for each worker / thread, using the py-tracebacker, I get no answer. The processes just hang.

How could I research what exact fact is what makes uwsgi processes hang?

How could I prevent this situation?

I know the harakiri parameter but I'm not sure if the process is killed if it has other active threads.

PD: "reload mercy" is set to a very high value avoid the killing of workers with still active threads (seems to be a bug). We have some Web requests which still take a long long time (which are in the way to be converted to jobs).

Thanks in advance.

Answer 1

Although I added already a comment, here goes a longer description.

Warning: the problem only arose when using more than one worker process AND more than one thread (-p --threads).

Short version: In Python 2.7.x some modules are not 100% thread safe during imports (logging, implicit imports of codecs). Try to import all of such problematic modules in the wsgi file given to uwsgi (i.e, before uwsgi forks).

Long version: In https://github.com/unbit/uwsgi/issues/1599 I analysed the problem and found that it could be related to python bug with the logging module. The problem can be solved importing and initializing any critical modules BEFORE uwsgi forks which happens after the wsgi script given to uwsgi is executed.

I finally resolved my problem importing the django settings.ROOT_URLCONF directly or indirectly from the wsgi file. This also has the benefits of reducing memory consumption (because the code base shared between workers is much bigger) and per-worker initialization time.