AttributeError: 'str' object has no attribute 'errno'

Question

I placed a ClientConnectionError exception in a multiprocessing.Queue that was generated by asyncio. I did this to pass an exception generated in asyncio land back to a client in another thread/process.

My assumption is that this exception occurred during the deserialization process reading the exception out of the queue. It looks pretty much impossible to reach otherwise.

Traceback (most recent call last):
  File "model_neural_simplified.py", line 318, in <module>
    main(**arg_parser())
  File "model_neural_simplified.py", line 314, in main
    globals()[command](**kwargs)
  File "model_neural_simplified.py", line 304, in predict
    next_neural_data, next_sample = reader.get_next_result()
  File "/project_neural_mouse/src/asyncs3/s3reader.py", line 174, in get_next_result
    result = future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "model_neural_simplified.py", line 245, in read_sample
    f_bytes = s3f.read(read_size)
  File "/project_neural_mouse/src/asyncs3/s3reader.py", line 374, in read
    size, b = self._issue_request(S3Reader.READ, (self.url, size, self.position))
  File "/project_neural_mouse/src/asyncs3/s3reader.py", line 389, in _issue_request
    response = self.communication_channels[uuid].get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.6/dist-packages/aiohttp/client_exceptions.py", line 133, in __init__
    super().__init__(os_error.errno, os_error.strerror)
AttributeError: 'str' object has no attribute 'errno'

I figure it's a long shot to ask, but does anyone know anything about this issue?

Python 3.6.8, aiohttp.__version__ == 3.6.0

Update:

I managed to reproduce the issue (credit to Samuel in comments for improving the minimal reproducible test case, and later xtreak at bugs.python.org for furthing distilling it to a pickle-only test case):

import pickle

ose = OSError(1, 'unittest')

class SubOSError(OSError):

    def __init__(self, foo, os_error):
        super().__init__(os_error.errno, os_error.strerror)

cce = SubOSError(1, ose)
cce_pickled = pickle.dumps(cce)
pickle.loads(cce_pickled)


./python.exe ../backups/bpo38254.py
Traceback (most recent call last):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/../backups/bpo38254.py", line 12, in <module>
    pickle.loads(cce_pickled)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/../backups/bpo38254.py", line 8, in __init__
    super().__init__(os_error.errno, os_error.strerror)
AttributeError: 'str' object has no attribute 'errno'

References:

• https://github.com/aio-libs/aiohttp/issues/4077
• https://bugs.python.org/issue38254

Answer 1

OSError has a custom __reduce__ implementation; unfortunately, it's not subclass friendly for subclasses that don't match the expected arguments. You can see the intermediate state of the pickling by calling __reduce__ manually:

>>> SubOSError.__reduce__(cce)
(modulename.SubOSError, (1, 'unittest'))

The first element of the tuple is the callable to call, the second is the tuple of arguments to pass. So when it tries to recreate your class, it does:

modulename.SubOSError(1, 'unittest')

having lost the information about the OSError you were originally created with.

If you must accept arguments that don't match what OSError.__reduce__/OSError.__init__ expects, you're going to need to write your own __reduce__ override to ensure the correct information is pickled. A simple version might be:

class SubOSError(OSError):

    def __init__(self, foo, os_error):
        self.foo = foo  # Must preserve information for pickling later
        super().__init__(os_error.errno, os_error.strerror)

    def __reduce__(self):
        # Pickle as type plus tuple of args expected by type
        return type(self), (self.foo, OSError(*self.args))

With that design, SubOSError.__reduce__(cce) would now return:

(modulename.SubOSError, (1, PermissionError(1, 'unittest')))

where the second element of the tuple is the correct arguments needed to recreate the instance (the change from OSError to PermissionError is expected; OSError actually returns its own subclasses based on the errno).