Gunicorn Django [CRITICAL] WORKER TIMEOUT

Question

Since I did a pip install google-api-python-client I have my Gunicorn workers stoping after timeout.

Django==1.5.3

Gunicorn==0.12.2

I'm not really sure if it comes from the pip but I did nothing particular except a database migration which migrated without error.

I use this command for Gunicorn:

gunicorn_django myapp.py --bind 127.0.0.1:8181 --timeout 120 --log-file /tmp/myapp.gunicorn.log --log-level info --workers 8 --pid /tmp/myapp.pid

I tryed the param --spew to have some trace but it doesn't help me:

[2016-06-13 21:09:52 +0000] [15602] [INFO] Worker exiting (pid: 15602)
[2016-06-13 21:09:52 +0000] [15601] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
    worker.init_process()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/app/djangoapp.py", line 106, in load
    return mod.make_wsgi_application()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/app/django_wsgi.py", line 37, in make_wsgi_application
    if get_validation_errors(s):
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/core/management/validation.py", line 35, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/db/models/loading.py", line 166, in get_app_errors
    self._populate()
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/db/models/loading.py", line 72, in _populate
    self.load_app(app_name, True)
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/db/models/loading.py", line 96, in load_app
    models = import_module('.models', app_name)
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/myapp/prod/apps/admin/models.py", line 5, in <module>
    from django.contrib.auth.models import User
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/contrib/auth/models.py", line 18, in <module>
    from django.contrib.auth.hashers import (
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/contrib/auth/hashers.py", line 8, in <module>
    from django.test.signals import setting_changed
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/test/__init__.py", line 6, in <module>
    from django.test.testcases import (TestCase, TransactionTestCase,
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/test/testcases.py", line 35, in <module>
    from django.test import _doctest as doctest
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/django/test/_doctest.py", line 104, in <module>
    import unittest, difflib, pdb, tempfile
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/pdbpp-0.7.2-py2.7.egg/pdb.py", line 38, in <module>
    pdb = import_from_stdlib('pdb')
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/pdbpp-0.7.2-py2.7.egg/pdb.py", line 35, in import_from_stdlib
    mydict = execfile(pyfile, result.__dict__)
  File "/usr/local/lib/python2.7/pdb.py", line 3, in <module>
    """A Python debugger."""
  File "/usr/local/lib/python2.7/pdb.py", line 3, in <module>
    """A Python debugger."""
  File "/home/myapp/.local/share/virtualenvs/myapp/lib/python2.7/site-packages/gunicorn/debug.py", line 40, in __call__
    line = src[lineno]
IndexError: tuple index out of range
[2016-06-13 21:09:52 +0000] [15601] [INFO] Worker exiting (pid: 15601)

As the problem came in the same time I installed google api client, I suspect pip to have upgraded some libs that are not compatible with my gunicorn or Django. I checked the pip log without success also.

If I run my Django app with runserver I can't see any bug, it seems very related to Gunicorn.

Is there a deeper way to debug Gunicorn ?

Answer 1

After struggling hours I finally found a clue in the pip log (HOME/.pip/pip.log) .

Installing google api client upgraded some of my previous libs like these:

Installing collected packages: pyopenssl, six, cryptography, idna, pyasn1, setuptools, enum34, ipaddress, cffi, pycparser
  Found existing installation: pyOpenSSL 0.14
    Uninstalling pyOpenSSL:
...
  Found existing installation: six 1.9.0
    Uninstalling six:
...
  Found existing installation: cryptography 0.7.1
    Uninstalling cryptography:

I noticed also some installing warning for cyptography. I decided to put back the old libs.

• pyOpenSSL 0.14
• six 1.9.0
• cryptography 0.7.1

And it solved the problem. I don't know if it is pyopenssl or cryptography but it is getting really boring to have all these libs problems.

Hope this will help someone next time.