Random database errors with Django 1.7, uwsgi and PostgreSQL

Question

After upgrading my Django 1.6 app to Django 1.7 I started to get random errors while fetching data from PostgreSQL:

DatabaseError: server sent data ("D" message) without prior row description ("T" message)
lost synchronization with server: got message type "�", length -1244613424

DatabaseError: lost synchronization with server: got message type "0", length 842674226

ProgrammingError: no results to fetch

ValueError: invalid literal for int() with base 10: 'feuj3f47jvsdv7tgnj43g63j'

When I quickly open 10 tabs in browser, half of the tabs load normally, half of them get a DB error. When I refresh the tabs where errors occurred, they load normally.

I'm running Django through uwsgi and nginx, the version of psycopg2 is 2.5.4.

Overall it looks like somehow communication with Postgres is completely broken and results of different queries get mixed.

---

Edit:

After several hours of troubleshooting I found out the following:

Django 1.6 + uwsgi - works
Django 1.7 + gunicorn - works
Django 1.7 + uwsgi - doesn't work, throws database errors. So the problem seems to be with specifically uwsgi and Django 1.7 combination. Which is strange, I have another Django 1.7 project running on the same server with the same uwsgi and it has no problems.

Any ideas?

(I don't really mind switching over to gunicorn, will probably have to go this way, but it's still interesting why is this happening)

---

Update 2: Closer examination shows completely insane things happening inside Django, like a model's primary key being substituted with current user's session_id (that's the source of "invalid literal for int() with base 10" error) and Django issuing queries to DB "forgetting" to specify WHERE clause. I would probably call that a memory corruption of some kind.

---

Update 3: We switched from uwsgi to gunicorn and the problems are now gone. Everything works great. I'm probably still looking for a proper solution though.

Answer 1

I think lazy-apps=true should do the trick. From uwsgi documentation:

uWSGI tries to (ab)use the Copy On Write semantics of the fork() call whenever possible. By default it will fork after having loaded your applications to share as much of their memory as possible. If this behavior is undesirable for some reason, use the lazy-apps option. This will instruct uWSGI to load the applications after each worker’s fork(). Beware as there is an older options named lazy that is way more invasive and highly discouraged (it is still here only for backward compatibility)

If you do not switch lazy-apps on, the workers will share memory, and most likely corrupt it:

a model's primary key being substituted with current user's session_id.

When it comes to connections and stuff, not setting lazy-apps is dangerous.

The drawback is that every worker will have a full connection pool (if connection pooling is taking place), and you might end up using a lot of connections.

I'm no python expert, but I think using something like gevent to handle the connection pooling in a centralized way. You might then even not need lazy-apps.