Why can't I catch this python exception? Exception module/class doesn't match the catched module/class

Question

I was writing some etcd modules for SaltStack and ran into this strange issue where it's somehow preventing me from catching an exception and I'm interested in how it's doing that. It seems specifically centered around urllib3.

A small script ( not salt ):

import etcd
c = etcd.Client('127.0.0.1', 4001)
print c.read('/test1', wait=True, timeout=2)

And when we run it:

[root@alpha utils]# /tmp/etcd_watch.py
Traceback (most recent call last):
  File "/tmp/etcd_watch.py", line 5, in <module>
    print c.read('/test1', wait=True, timeout=2)
  File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
    timeout=timeout)
  File "/usr/lib/python2.6/site-packages/etcd/client.py", line 788, in api_execute
    cause=e
etcd.EtcdConnectionFailed: Connection to etcd failed due to ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.",)

Ok, let's catch that bugger:

#!/usr/bin/python

import etcd
c = etcd.Client('127.0.0.1', 4001)

try:
  print c.read('/test1', wait=True, timeout=2)
except etcd.EtcdConnectionFailed:
  print 'connect failed'

Run it:

[root@alpha _modules]# /tmp/etcd_watch.py
connect failed

Looks good - it's all working python. So what's the issue? I have this in the salt etcd module:

[root@alpha _modules]# cat sjmh.py
import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    return c.read('/test1', wait=True, timeout=2)
  except etcd.EtcdConnectionFailed:
    return False

And when we run that:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1173, in _thread_return
        return_data = func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/sjmh.py", line 5, in test
        c.read('/test1', wait=True, timeout=2)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
        timeout=timeout)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 769, in api_execute
        _ = response.data
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 150, in data
        return self.read(cache_content=True)
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 218, in read
        raise ReadTimeoutError(self._pool, None, 'Read timed out.')
    ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.

Hrm, that's weird. etcd's read should have returned etcd.EtcdConnectionFailed. So, let's look at it further. Our module is now this:

import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    return c.read('/test1', wait=True, timeout=2)
  except Exception as e:
    return str(type(e))

And we get:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    <class 'urllib3.exceptions.ReadTimeoutError'>

Ok, so we know that we can catch this thing. And we now know it threw a ReadTimeoutError, so let's catch that. The newest version of our module:

import etcd
import urllib3.exceptions

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError as e:
    return 'caught ya!'
  except Exception as e:
    return str(type(e))

And our test..

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    <class 'urllib3.exceptions.ReadTimeoutError'>

Er, wait, what? Why didn't we catch that? Exceptions work, right.. ?

How about if we try and catch the base class from urllib3..

[root@alpha _modules]# cat sjmh.py
import etcd
import urllib3.exceptions

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.HTTPError:
    return 'got you this time!'

Hope and pray..

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/salt/minion.py", line 1173, in _thread_return
        return_data = func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/sjmh.py", line 7, in test
        c.read('/test1', wait=True, timeout=2)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 481, in read
        timeout=timeout)
      File "/usr/lib/python2.6/site-packages/etcd/client.py", line 769, in api_execute
        _ = response.data
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 150, in data
        return self.read(cache_content=True)
      File "/usr/lib/python2.6/site-packages/urllib3/response.py", line 218, in read
        raise ReadTimeoutError(self._pool, None, 'Read timed out.')
    ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=4001): Read timed out.

BLAST YE! Ok, let's try a different method that returns a different etcd Exception. Our module now looks like this:

import etcd

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.delete('/')
  except etcd.EtcdRootReadOnly:
    return 'got you this time!'

And our run:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    got you this time!

As a final test, I made this module, which I can run either from straight python, or as a salt module..

import etcd
import urllib3

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError:
    return 'got you this time!'
  except etcd.EtcdConnectionFailed:
    return 'cant get away from me!'
  except etcd.EtcdException:
    return 'oh no you dont'
  except urllib3.exceptions.HTTPError:
    return 'get back here!'
  except Exception as e:
    return 'HOW DID YOU GET HERE? {0}'.format(type(e))

if __name__ == "__main__":
  print test()

Through python:

[root@alpha _modules]# python ./sjmh.py
cant get away from me!

Through salt:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    HOW DID YOU GET HERE? <class 'urllib3.exceptions.ReadTimeoutError'>

So, we can catch exceptions from etcd that it throws. But, while we normally are able to catch the urllib3 ReadTimeoutError when we run python-etcd by its lonesome, when I run it through salt, nothing seems to be able to catch that urllib3 exception, except a blanket 'Exception' clause.

I can do that, but I'm really curious as to what the heck salt is doing that's making it so that an exception is uncatchable. I've never seen this before when working with python, so I'd be curious as to how it's happening and how I can work around it.

Edit:

So I was finally able to catch it.

import etcd
import urllib3.exceptions
from urllib3.exceptions import ReadTimeoutError

def test():
  c = etcd.Client('127.0.0.1', 4001)
  try:
    c.read('/test1', wait=True, timeout=2)
  except urllib3.exceptions.ReadTimeoutError:
    return 'caught 1'
  except urllib3.exceptions.HTTPError:
    return 'caught 2'
  except ReadTimeoutError:
    return 'caught 3'
  except etcd.EtcdConnectionFailed as ex:
    return 'cant get away from me!'
  except Exception as ex:
    return 'HOW DID YOU GET HERE? {0}'.format(type(ex))

if __name__ == "__main__":
  print test()

And when run:

[root@alpha _modules]# salt 'alpha' sjmh.test
alpha:
    caught 3

It still doesn't make sense though. From what I know of exceptions, the return should be 'caught 1'. Why should I have to import the name of the exception directly, rather than just using full class name?

MORE EDITS!

So, adding the comparison between the two classes produces 'False' - which is sorta obvious, because the except clause wasn't working, so those couldn't be the same.

I added the following to the script, right before I call the c.read().

log.debug(urllib3.exceptions.ReadTimeoutError.__module__)
log.debug(ReadTimeoutError.__module__)

And now I get this in the log:

[DEBUG   ] requests.packages.urllib3.exceptions
[DEBUG   ] urllib3.exceptions

So, that appears to be the reason that is getting caught the way it is. This is also reproducible by just downloading the etcd and requests library and doing something like this:

#!/usr/bin/python

#import requests
import etcd

c = etcd.Client('127.0.0.1', 4001)
c.read("/blah", wait=True, timeout=2)

You'll end up getting the 'right' exception raised - etcd.EtcdConnectionFailed. However, uncomment 'requests' and you'll end up with urllib3.exceptions.ReadTimeoutError, because etcd now no longer catches the exception.

So it appears that when requests is imported, it rewrites the urllib3 exceptions, and any other module that is trying to catch those, fails. Also, it appears that newer versions of requests do not have this issue.

Answer 1

My answer below is a little of speculation, because I cannot prove it on practice with these exact libraries (to start with I cannot reproduce your error as it also depends on libraries versions and how they are installed), but nevertheless shows one of the possible ways of this happening:

The very last example gives a good clue: the point is indeed that at different moments in the time of the program execution, name urllib3.exceptions.ReadTimeoutError may refer to different classes. ReadTimeoutError is, just like for every other module in Python, is simply a name in the urllib3.exceptions namespace, and can be reassigned (but it doesn't mean it is a good idea to do so).

When referring to this name by its fully-qualified "path" - we are guaranteed to refer to the actual state of it by the time we refer to it. However, when we first import it like from urllib3.exceptions import ReadTimeoutError - it brings name ReadTimeoutError into the namespace which does the import, and this name is bound to the value of urllib3.exceptions.ReadTimeoutError by the time of this import. Now, if some other code reassigns later the value for urllib3.exceptions.ReadTimeoutError - the two (its "current"/"latest" value and the previously imported one) might be actually different - so technically you might end up having two different classes. Now, which exception class will be actually raised - this depends on how the code which raises the error uses it: if they previously imported the ReadTimeoutError into their namespace - then this one (the "original") will be raised.

To verify if this is the case you might add the following to the except ReadTimeoutError block:

print(urllib3.exceptions.ReadTimeoutError == ReadTimeoutError)

If this prints False - it proves that by the time the exception is raised, the two "references" refer to different classes indeed.

---

A simplified example of a poor implementation which can yield similar result:

File api.py (properly designed and exists happily by itself):

class MyApiException(Exception):
    pass

def foo():
    raise MyApiException('BOOM!')

File apibreaker.py (the one to blame):

import api

class MyVeryOwnException(Exception):
    # note, this doesn't extend MyApiException,
    # but creates a new "branch" in the hierarhcy
    pass

# DON'T DO THIS AT HOME!
api.MyApiException = MyVeryOwnException

File apiuser.py:

import api
from api import MyApiException, foo
import apibreaker

if __name__ == '__main__':
    try:
        foo()
    except MyApiException:
        print("Caught exception of an original class")
    except api.MyApiException:
        print("Caught exception of a reassigned class")

When executed:

$ python apiuser.py
Caught exception of a reassigned class

If you remove the line import apibreaker - clearly then all goes back to their places as it should be.

This is a very simplified example, yet illustrative enough to show that when a class is defined in some module - newly created type (object representing new class itself) is "added" under its declared class name to the module's namespace. As with any other variable - its value can be technically modified. Same thing happens to functions.