isinstance() unexpectedly returning False

Question

I am using the kubernetes python client. In the event that kubernetes isn't available when my code starts up, I would like to retry the connection.

When the client is unable to connect, it throws what appears to be a urllib3.exceptions.MaxRetryError exception, so I started with something like this:

import time
import urllib3

import kubernetes

kubernetes.config.load_kube_config()
api = kubernetes.client.CoreV1Api()

while True:
    try:
        w = kubernetes.watch.Watch()
        for event in w.stream(api.list_pod_for_all_namespaces):
            print event
    except urllib3.exceptions.HTTPError:
        print('retrying in 1 second')
        time.sleep(1)

But that completely fails; it acts like there is no except statement and bails out with:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2743110>: Failed to establish a new connection: [Errno 111] Connection refused',))

I thought maybe I didn't understand inheritance as well as I thought, so I replace the above with:

except urllib3.exceptions.MaxRetryError:
    print('retrying in 1 second')
    time.sleep(1)

Which fails in the same way. In an effort to figure out what was going on, I added a catch-all except and invoked pdb:

except Exception as err:
    import pdb; pdb.set_trace()

And from the pdb prompt, we can see:

(Pdb) type(err)
<class 'urllib3.exceptions.MaxRetryError'>

...which looks fine, as does the mro:

(Pdb) import inspect
(Pdb) inspect.getmro(err.__class__)
(<class 'urllib3.exceptions.MaxRetryError'>, <class 'urllib3.exceptions.RequestError'>, <class 'urllib3.exceptions.PoolError'>, <class 'urllib3.exceptions.HTTPError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>)

But despite all that:

(Pdb) isinstance(err, urllib3.exceptions.MaxRetryError)
False

And all the paths look reasonable:

(Pdb) urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
(Pdb) kubernetes.client.rest.urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'

So...what the actual what is going on here?

Update

Here is the full stack trace:

Traceback (most recent call last):
  File "testkube.py", line 13, in <module>
    for event in w.stream(api.list_pod_for_all_namespaces):
  File "/usr/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 116, in stream
    resp = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14368, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14464, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
    _preload_content, _request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
    query_params=query_params)
  File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 223, in request
    headers=headers)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x3d16110>: Failed to establish a new connection: [Errno 111] Connection refused',))

Answer 1

Your code has an indirect dependency on the requests package, and the requests package has a strange submodule called requests.packages. This used to contain copied source code from a number of dependencies, including urllib3, but they stopped doing that. They wanted to keep requests.packages around for backward compatibility, though, so now they're doing something weird.

Instead of requests.packages including a complete copy of the urllib3 source code, it now imports urllib3 and sets sys.modules['requests.packages.urllib3'] = urllib3. Depending on the requests version, it may set a number of other sys.modules entries, too; for example, as of requests 2.18.4, the source code does

for package in ('urllib3', 'idna', 'chardet'):
    locals()[package] = __import__(package)
    # This traversal is apparently necessary such that the identities are
    # preserved (requests.packages.urllib3.* is urllib3.*)
    for mod in list(sys.modules):
        if mod == package or mod.startswith(package + '.'):
            sys.modules['requests.packages.' + mod] = sys.modules[mod]

but in 2.17.0, it does

import urllib3
sys.modules['requests.packages.urllib3'] = urllib3

import idna
sys.modules['requests.packages.idna'] = idna

import chardet
sys.modules['requests.packages.chardet'] = chardet

This code interacts badly with submodules of the imported packages. If some code tries to do import requests.packages.urllib3.exceptions and Python doesn't find a sys.modules['requests.packages.urllib3.exceptions'] entry, Python will recreate the urllib3.exceptions module and set urllib3.exceptions and sys.modules['requests.packages.urllib3.exceptions'] to the new module (but it won't touch sys.modules['urllib3.exceptions']. This will generate new copies of the classes involved, causing your error.

A related problem with the same cause was reported back in May, leading to the new code shown in 2.18.4. 2.18.4 shouldn't cause the specific problems you're seeing, but it's still fragile, because if any submodules of urllib3 aren't yet loaded at the time requests.packages screws with sys.modules, those submodules will exhibit the same problems you've seen today.