cherrypy /dev/urandom (or equivalent) not found -- error

Question

I am running a cherrypy 3.2.0 server with Python 2.5.1, which gives the following error every few days on any instruction from UI until it is killed and re-started:-

[29/Mar/2012:06:37:57] HTTP Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/_cprequest.py", line 636, in respond
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/_cprequest.py", line 97, in run
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/_cprequest.py", line 57, in __call__
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/lib/sessions.py", line 757, in init
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/lib/sessions.py", line 162, in __init__
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/lib/sessions.py", line 190, in _regenerate
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/lib/sessions.py", line 204, in generate_id
File "/usr/lib/python2.5/site-packages/CherryPy-3.2.0-py2.5.egg/cherrypy/_cpcompat.py", line 264, in random20
File "/usr/lib/python2.5/os.py", line 733, in urandom
NotImplementedError: /dev/urandom (or equivalent) not found

_cpcompat.py has following piece of code which suggests that there is a fallback on random.random in case cherrypy is not able to read /dev/urandom, but doesn't seem to be falling back on it.

try:
    os.urandom(20)
    import binascii
    def random20():
        return binascii.hexlify(os.urandom(20)).decode('ascii')

except (AttributeError, NotImplementedError):
    import random
    # os.urandom not available until Python 2.4. Fall back to random.random.
    def random20(): 
        return sha('%s' % random.random()).hexdigest()

Following is the code snippet from os.py, relevant in the context:-

if not _exists("urandom"):

    def urandom(n):
        """urandom(n) -> str

        Return a string of n random bytes suitable for cryptographic use.

        """
        try:
            _urandomfd = open("/dev/urandom", O_RDONLY)
        except (OSError, IOError):
            raise NotImplementedError("/dev/urandom (or equivalent) not found")
        bytes = ""
        while len(bytes) < n:
            bytes += read(_urandomfd, n - len(bytes))
        close(_urandomfd)
        return bytes

At the same time when cherrypy is not able to read /dev/urandom, the following code snippet is working fine :-

python -c "import os;fd = open('/dev/urandom', 'r');print fd.read(5);fd.close()"

I have two questions:-

1. Why is cherrypy throwing not implemented error when I am able to read random bits from /dev/urandom
2. Why is _cpcompact.py not executing the except part when os.py is raising NotImplementedError.

Answer 1

This is not an answer either, however, in my experience this NotImplementedError is erroneous but shows up when 'too many files' are open errors start showing up after a multiprocessing Processes start throwing uncaught exception, left to dangle forever in child threads or child processes.

I would start debugging further up the stack and see if there are buried or silent exceptions thrown by a Process

Here's an example of my stack trace when I start seeing this error

Exception in thread Plotter:
Traceback (most recent call last):
  File "threading.pyc", line 532, in __bootstrap_inner
  File "plotters/edge.pyc", line 459, in run
  AttributeError: 'error' object has no attribute 'error'

OSError: [Errno 24] Too many open files
  File "multiprocessing/connection.pyc", line 150, in Client
  File "multiprocessing/connection.pyc", line 370, in deliver_challenge
    None
  File "os.pyc", line 756, in urandom
NotImplementedError: /dev/urandom (or equivalent) not found

My AttributeError produced, eventually... the /urandom not found imlp error

Answer 2

This isn't an answer, but maybe you could put some debug code into os.py (I can't imagine it would impact any other program using import os but its worth remembering its customized)

if not _exists("urandom"):
    def urandom(n):
        """urandom(n) -> str

        Return a string of n random bytes suitable for cryptographic use.

        """
        try:
            _urandomfd = open("/dev/urandom", O_RDONLY)
        # debug changes
        except (OSError, IOError) as Err:
            import syslog
            syslog.syslog(repr(Err))
        # /debug
            raise NotImplementedError("/dev/urandom (or equivalent) not found")
        bytes = ""
        while len(bytes) < n:
            bytes += read(_urandomfd, n - len(bytes))
        close(_urandomfd)
        return bytes

Hopefully that would tell you exactly what the error is. (You can of course replace syslog with just writing to a file etc)