AppEngine urlfetch validate_certificate=False/None not being respected

Question

In the AppEngine developer appserver I am getting an error like this:

SSLCertificateError: Invalid and/or missing SSL certificate for URL ...

when I am making a fetch like this to an https server with a self-signed certificate (almost always localhost port-forwarded over ssh to a vm):

result = urlfetch.fetch(url=url, method=method, payload=payload,
                        deadline=DEADLINE, validate_certificate=None)

One would not expect SSL failures for invalid certificates where validate_certificate is False, though this is quite possibly a side-effect of the 2.7.9 policy in Python to always validate ssl certificates.

Note that passing False (instead of None) for validate_certificate does not work either.

This problem happens on Python 2.7.9-10 via Homebrew/XCode on OS X 10.10.2-4 with AppEngine 1.9.18 through 1.19.26.

There are issues (e.g. 12096) about this on Google App Engine, but I am looking for a workaround.

Here's what I've tried to work around this:

1. Add the certificate to the Mac's login keychain (works in the browser, not from Python)

2. Add the certificate to app-engine-python/lib/cacerts/cacerts.txt and/or ./lib/cacerts/urlfetch_cacerts.txt (though this probably requires turning verification on for it to work, since that appears to be the only case where they are used) with e.g.

   $ echo >> /usr/local/share/app-engine-python/lib/cacerts/urlfetch_cacerts.txt

   $ openssl x509 -subject -in server.crt >> /usr/local/share/app-engine-python/lib/cacerts/urlfetch_cacerts.txt

3. Disable ssl HTTPs checking with the PEP-0476 workaround i.e.

   ssl._create_default_https_context = ssl._create_unverified_context

   at or after import ssl (around line 1149) of google/appengine/dist27/python_std_lib/httplib.py

This is particularly problematic on Mac since downgrading as of XCode 7/OS X El Capital is no longer a practical option.

A preferable workaround would not involve monkey-patching the AppEngine code proper every time the development appserver is updated.

---

EDIT

Note that the Mac builtin OpenSSL certificates are stored in /System/Library/OpenSSL, which is protected with SIP/rootlessness, which frankly is a pain to muck with and a worthwhile feature to keep if we can.

I have verified that the certificate validates by using openssl s_client -connect localhost:7500 -CAfile server.pem.

It's been added to the Keychain and to /usr/local/etc/openssl/certs with the hash.# format where the hash comes from openssl x509 -subject_hash -in server.pem (or the homebrew ssl, namely /usr/local/Cellar/openssl/1.0.2d_1/bin/openssl). In which case /usr/local/Cellar/openssl/1.0.2d_1/bin/openssl s_client -connect localhost:7500 verifies the certificate (but python still does not).

I have tried using the homebrew version of python and openssl, but to no avail. Running the following in Python seems to always fail;

./pve/bin/python -c "import requests; requests.get('https://localhost:7500')"

This also fails where SSL_CERT_FILE is set to the server's certificate (i.e. for added measure one might expect it to work since the openssl command essentially works like this), and also fails where SSL_CERT_PATH is set to /usr/local/etc/openssl/certs.

Note, pve is a virtual env where help(ssl) shows a FILE of /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py

Further verifying that homebrew Python's _ssl.so links to homebrew's openssl I ran:

xcrun otool -L /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so

which returns

./Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so:

/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)

If one runs brew info openssl it remarks under CAVEATS:

A CA file has been bootstrapped using certificates from the system keychain. To add additional certificates, place .pem files in /usr/local/etc/openssl/certs

but clearly for some reason python is not using homebrew's openssl algorithm for finding certificates.

So I remain at a loss as to why Python standard library is not validating certificates that are in the OpenSSL directory specified in the documents as well as the Keychain (in both .pem and .p12 formats, with "always trust" for Secure Sockets Layer (SSL)).

Answer 1

This is a dev_appserver bug caused by a httplib.HTTPSConnection behavior change (certificate check turned on by default) in some recent Python release (I belive 2.7.9).

As the bug is in internal dev_appserver code (file google_appengine/google/appengine/api/urlfetch_stub.py of the appengine SDK) that is run independently of the tested application, there is no way to make a fix that will survive a SDK update.

The only permanent workaround I can think of will be to enable validate_certificate and add CA certificate to the urlfetch_cacerts.txt file. As a temporary fix, you can patch urlfetch_stub.py with workaround #3.