Python Requests SSLError with internal CA

Question

My company operates its own internal CA for internal services, and I need to access hook up Ansible AWX [python] to talk to one of our internal services which uses a cert signed by this CA. Basically:

1. AWX spins up a container awx_task with /etc/pki/ca-trust/source/anchors mounted in, which contains the root CA cert. [double-checked]
2. update-ca-trust is run, bundling the CA cert into various things, including /etc/pki/tls/certs/ca-bundle.crt. [double-checked]
3. requests should use this bundle. There are no CA-related environment variables that I can find inside the container or on the host that would override this.

However when I trigger a test run of an Ansible play which runs inside of awx_task I get the error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='vault.example.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))

On the host machine I can run

import requests
requests.get("https://vault.example.com")

and get a 200 response, and if I strace the process I can see it reading /etc/pki/tls/certs/ca-bundle.crt. But from inside awx_task I get the same requests.exceptions.SSLError as above. Unfortunately Docker won't let me run strace inside the container so I can't see what it's trying to read.

But if I modify the code to:

import requests
requests.get("https://vault.example.com", verify="/etc/pki/tls/certs/ca-bundle.crt")

I get a 200 response from inside the container.

What am I missing here?

Answer 1

The problem is what @Will noted, the current version of Requests uses the Certifi bundle which is entirely separate from OpenSSL. The bundle PEM actually lives somewhere in you Python site-packages dir.

Without modifying your code you can override this with the environment variable:

REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt

Editorial: This is an absolutely ridiculous way to enforce CA trust. If you want to pare down your system trust, pare it down at the system level. I'm getting real sick of chasing down random PEM bundles scattered through source trees [which probably never get updated] just because some #devoops nutbag thinks he knows how to run systems better than actual ops and forking their bad ideas off to unsuspecting systems.

(ノಠ益ಠ)ノ彡┻━┻