Intermittent "getrandom() initialization failed" using scrapy spider

Question

I built a scrapy spider (scrapy 1.4). This spider is triggered on demand from a django website through django-rq and supervisord.

Here is the supervisord job that is listening for django-rq events (reddit is used as broker)

[program:rq_worker] 
command=python3 manage.py rqworker default 
directory=/var/www/django-app 
autostart=true 
autorestart=true 
stderr_logfile=/var/log/rq_worker.err.log 
stdout_logfile=/var/log/rq_worker.out.log

This set up is running fine. However, from time to time (I cannot reproduce the issue on demand), all the spiders throw the same OpenSSL error:

2018-02-11 11:02:19 [scrapy.core.scraper] ERROR: Error downloading <GET https://whateverwebsite.com>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.5/dist-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.5/dist-packages/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('', 'osrandom_rand_bytes', 'getrandom() initialization failed.')]>]

Restarting supervisord makes the issue disappearing.

To make sure my website and its spiders are running properly I have to test each time supervisord is restarted that there is no issue. Not a big deal but still...

I would like to understand what's going wrong there? How can I troubleshoot this issue? Is it supervisord related? Twisted related? openSSL related?

Thank you for your help

Answer 1

I had similar error, but with python-requests library:

Error([('', 'osrandom_rand_bytes', 'getrandom() initialization failed.')],)

This was caused by random number generator that failed to gather enough entropy in time. I've installed rng-tools and it solved the problem.