Gunicorn exits because https request from Django fails in OpenSSL

Question

Here's the scenario. I have a Django app being served by Gunicorn on Linux. On certain requests it makes an https call to an external API via httplib2.request(). Sometimes that call fails in such a way that hoses OpenSSL (not sure how, but it's not my fault and doesn't really matter). OpenSSL sends a SIGABRT signal to gunicorn in this case. Gunicorn handles the SIGABRT and promptly system exits (as it should).

The root issue as I see it is that OpenSSL asynchronously signals the parent process to abort, rather than returning an error code. Don't tell me to abort because of YOUR personal problems, OpenSSL! Legacy code in action.

Is anyone else thoroughly annoyed by this problem? How would you prevent this from killing your Gunicorn process? It completely bypasses Django exception handling.

Relevant code points:

• OpenSSL sends SIGABRT whenever OpenSSLDie() is called: https://github.com/openssl/openssl/blob/e0fc7961c4fbd27577fb519d9aea2dc788742715/crypto/cryptlib.c#L391
• which comments claim happens whenever there is a fatal error in any cryptographic operation: https://github.com/openssl/openssl/blob/e0fc7961c4fbd27577fb519d9aea2dc788742715/fips/fips.c#L136
• The SIGABRT is handled by gunicorn: https://github.com/benoitc/gunicorn/blob/6eb01409da42a81b7020cd78c52613d8ec868e94/gunicorn/workers/base.py#L173
• which causes gunicorn to sys.exit(1): https://github.com/benoitc/gunicorn/blob/6eb01409da42a81b7020cd78c52613d8ec868e94/gunicorn/workers/base.py#L200

Answer 1

I don't think the signal gets created by OpenSSL. The signal comes from gunicorn which sends this signal if the request takes too long.

worker_abort(worker): Called when a worker received the SIGABRT signal.

This call generally happens on timeout.

The callable needs to accept one instance variable for the initialized Worker.

https://docs.gunicorn.org/en/stable/settings.html#worker-abort

My traceback, which I get if the request takes more than N seconds:

SystemExit: 1
  File "django/core/handlers/wsgi.py", line 133, in __call__
    response = self.get_response(request)
  File "django/core/handlers/base.py", line 128, in get_response
    response = self._middleware_chain(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "debug_toolbar/middleware.py", line 48, in __call__
    return self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "whitenoise/middleware.py", line 59, in __call__
    response = self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/utils/deprecation.py", line 114, in __call__
    response = response or self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django_middleware_global_request/middleware.py", line 15, in __call__
    return self.get_response(request)
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/contrib/admin/options.py", line 614, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "django/contrib/admin/options.py", line 1719, in changelist_view
    response = self.response_action(request, queryset=cl.get_queryset(request))
  File "django/contrib/admin/options.py", line 1402, in response_action
    response = func(self, request, queryset)
  File "lala/admin.py", line 252, in send_invoice_via_mail
    send_invoice_mail_failsafe_and_create_log(invoice)
  File "lala/views/send_invoices_via_mail.py", line 24, in send_invoice_mail_failsafe_and_create_log
    return send_invoice_mail(invoice)
  File "contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "lala/views/send_invoices_via_mail.py", line 39, in send_invoice_mail
    mail.send_mail(
  File "django/core/mail/__init__.py", line 61, in send_mail
    return mail.send()
  File "django/core/mail/message.py", line 284, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "django/core/mail/backends/smtp.py", line 109, in send_messages
    sent = self._send(message)
  File "django/core/mail/backends/smtp.py", line 125, in _send
    self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n'))
  File "smtplib.py", line 865, in sendmail
    (code, resp) = self.mail(from_addr, esmtp_opts)
  File "smtplib.py", line 539, in mail
    return self.getreply()
  File "smtplib.py", line 391, in getreply
    line = self.file.readline(_MAXLINE + 1)
  File "socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
  File "gunicorn/workers/base.py", line 201, in handle_abort
    sys.exit(1)