Python requests SSL certificate verification fails, even after adding CA certificates

Question

I've been using the Python Requests library to scrape a website for a while now, but the website recently changed SSL certificates, and the new cert won't verify with requests.

Based on answers to similar questions, I've updated requests and urllib3 to the latest versions (2.4.3 and 1.9.1), and manually added the CA certs to requests' cacert.pem (/usr/local/lib/python2.7/dist-packages/requests/cacert.pem).

I can successfully use this cacert.pem file with curl, but still not with requests:

> curl --head --cacert /usr/local/lib/python2.7/dist-packages/requests/cacert.pem
https://jordan-cu.org

HTTP/1.1 200 OK
Date: Thu, 20 Nov 2014 16:21:28 GMT
Server: Apache
X-Pingback: https://jordan-cu.org/xmlrpc.php
Link: <https://jordan-cu.org/>; rel=shortlink
X-Powered-By: PleskLin
Content-Type: text/html; charset=UTF-8

> python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> requests.get('https://jordan-cu.org')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I'm not sure what else to try at this point. Any help is appreciated!

Answer 1

You need to install pyopenssl and ndg-httpsclient

See using requests with TLS doesn't give SNI support for more details

Answer 2

Python2 has no support for SNI and requests does not help in this regard, see http://docs.python-requests.org/en/latest/community/faq/. But, if accessed without SNI the server returns a self-signed certificate:

$ openssl s_client -connect jordan-cu.org:443 | openssl x509 -text -noout
...
Issuer: C=US, ST=Virginia, L=Herndon, O=Parallels, OU=Parallels Panel, CN=Parallels Panel/[email protected]
...
Subject: C=US, ST=Virginia, L=Herndon, O=Parallels, OU=Parallels Panel, CN=Parallels Panel/[email protected]

If one instead accesses the server with SNI it returns a certificate signed by a public CA:

$ openssl s_client -connect jordan-cu.org:443 -servername jordan-cu.org | openssl x509 -text -noout
...
Issuer: C=US, O=GeoTrust, Inc., CN=RapidSSL CA
...
Subject: ... CN=*.jordan-cu.org

Because Python2 does the TLS connection without SNI you will get the self-signed certificate which of course can not be verified against cacert.pem and thus you get certificate verify failed.

Fix: Upgrade to Python3 which supports SNI.