uWSGI + nginx for django app avoids pylibmc multi-thread concurrency issue?

Question

Introduction

I encountered this very interesting issue this week, better start with some facts:

• pylibmc is not thread safe, when used as django memcached backend, starting multiple django instance directly in shell would crash when hit with concurrent requests.
• if deploy with nginx + uWSGI, this problem with pylibmc magically dispear.
• if you switch django cache backend to python-memcached, it too will solve this problem, but this question isn't about that.

Elaboration

start with the first fact, this is how I reproduced the pylibmc issue:

The failure of pylibmc

I have a django app which does a lot of memcached reading and writing, and there's this deployment strategy, that I start multiple django process in shell, binding to different ports (8001, 8002), and use nginx to do the balance.

I initiated two separate load test against these two django instance, using locust, and this is what happens:

In the above screenshot they both crashed and reported exactly the same issue, something like this:

Assertion "ptr->query_id == query_id +1" failed for function "memcached_get_by_key" likely for "Programmer error, the query_id was not incremented.", at libmemcached/get.cc:107

uWSGI to the rescue

So in the above case, we learned that multi-thread concurrent request towards memcached via pylibmc could cause issue, this somehow doesn't bother uWSGI with multiple worker process.

To prove that, I start uWSGI with the following settings included:

master          = true
processes       = 2

This tells uWSGI to start two worker process, I then tells nginx to server any django static files, and route non-static requests to uWSGI, to see what happens. With the server started, I launch the same locust test against django in localhost, and make sure there's enough requests per seconds to cause concurrent request against memcached, here's the result:

In the uWSGI console, there's no sign of dead worker processes, and no worker has been re-spawn, but looking at the upper part of the screenshot, there sure has been concurrent requests (5.6 req/s).

The question

I'm extremely curious about how uWSGI make this go away, and I couldn't learn that on their documentation, to recap, the question is:

How did uWSGI manage worker process, so that multi-thread memcached requests didn't cause django to crash?

In fact I'm not even sure that it's the way uWSGI manages worker processes that avoid this issue, or some other magic that comes with uWSGI that's doing the trick, I've seen something called a memcached router in their documentation that I didn't quite understand, does that relate?

Answer 1

Isn't it because you actually have two separate processes managed by uWSGI? As you are setting the processes option instead of the workers option, so you should actually have multiple uWSGI processes (I'm assuming a master + two workers because of the config you used). Each of those processes will have it's own loaded pylibmc, so there is not state sharing between threads (you haven't configured threads on uWSGI after all).