openssl, python requests error: "certificate verify failed"

Question

If I run the following command from my development box:

$ openssl s_client -connect github.com:443 

I get the following last line of output:

Verify return code: 20 (unable to get local issuer certificate) 

If I try to do this with requests I get another failed request:

>>> import requests >>> r = requests.get('https://github.com/', verify=True) 

With an exception raised:

SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed 

I can also run the first command with the verify flag and get similar output:

$ openssl s_client -connect github.com:443 -verify 9 ... Verify return code: 27 (certificate not trusted) 

Basically this is telling me that there is a problem with the certificates. I can specify a specific certificate with both methods and it will work:

$ openssl s_client -connect github.com:443 -CAfile /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -verify 9 ... Verify return code: 0 (ok) 

and:

>>> r = requests.get('https://github.com/', verify='/etc/ssl/certs/DigiCert...pem') <Response [200]> 

So, to my question, what exactly is wrong here? Shouldn't requests/openssl already know where to find valid certs?

Other Info:

• Python==2.7.6
• requests==2.2.1
• openssl 0.9.8h

Also, I know passing verify=False to the requests.get method will work too, but I do want to verify.

EDIT

I've confirmed that, as @Heikki Toivonen indicated in an answer, specifying the -CAfile flag for the version of openssl that I'm running works.

$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'` ... Verify return code: 0 (ok) 

So there is nothing wrong with the version of openssl that I'm running, and there is nothing wrong with the default cacert.pem file that requests provides.

Now that I know openssl is meant to work that way, that the CAfile or the place to find certs has to be specified, I'm more concerned about getting requests to work.

If I run:

>>> r = requests.get('https://github.com/', verify='path to cacert.pem file') 

I'm still getting the same error as before. I even tried downloading the cacert.pem file from http://curl.haxx.se/ca and it still didn't work. requests only seems to work (on this specific machine) if I specify a specific vendor cert file.

A side note: On my local machine everything is working as expected. There are several difference between the two machines though. I so far haven't been able to determine what the specific difference is that causes this issue.

Answer 1

If I run the following command from my development box:

$ openssl s_client -connect github.com:443 

I get the following last line of output:

Verify return code: 20 (unable to get local issuer certificate) 

You are missing DigiCert High Assurance EV CA-1 as a root of trust:

$ openssl s_client -connect github.com:443 CONNECTED(00000003) depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1 verify error:num=20:unable to get local issuer certificate verify return:0 --- Certificate chain  0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com    i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1  1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1    i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA --- Server certificate ... Start Time: 1393392088 Timeout   : 300 (sec) Verify return code: 20 (unable to get local issuer certificate) 

Download DigiCert High Assurance EV CA-1 from DigiCert Trusted Root Authority Certificates:

$ wget https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt --2014-02-26 00:27:50--  https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt Resolving www.digicert.com (www.digicert.com)... 64.78.193.234 ... 

Convert the DER encoded certifcate to PEM:

$ openssl x509 -in DigiCertHighAssuranceEVCA-1.crt -inform DER -out DigiCertHighAssuranceEVCA-1.pem -outform PEM 

Then, use it with OpenSSL via the -CAfile:

$ openssl s_client -CAfile DigiCertHighAssuranceEVCA-1.pem -connect github.com:443 CONNECTED(00000003) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1 verify return:1 depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 5157550, street = 548 4th Street, postalCode = 94107, C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com verify return:1 --- Certificate chain  0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com    i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1  1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1    i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA --- Server certificate -----BEGIN CERTIFICATE----- MIIHOjCCBiKgAwIBAgIQBH++LkveAITSyvjj7P5wWDANBgkqhkiG9w0BAQUFADBp MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3 d3cuZGlnaWNlcnQuY29tMSgwJgYDVQQDEx9EaWdpQ2VydCBIaWdoIEFzc3VyYW5j ZSBFViBDQS0xMB4XDTEzMDYxMDAwMDAwMFoXDTE1MDkwMjEyMDAwMFowgfAxHTAb BgNVBA8MFFByaXZhdGUgT3JnYW5pemF0aW9uMRMwEQYLKwYBBAGCNzwCAQMTAlVT MRkwFwYLKwYBBAGCNzwCAQITCERlbGF3YXJlMRAwDgYDVQQFEwc1MTU3NTUwMRcw FQYDVQQJEw41NDggNHRoIFN0cmVldDEOMAwGA1UEERMFOTQxMDcxCzAJBgNVBAYT AlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv MRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xEzARBgNVBAMTCmdpdGh1Yi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDt04nDXXByCfMzTxpydNm2WpVQ u2hhn/f7Hxnh2gQxrxV8Gn/5c68d5UMrVgkARWlK6MRb38J3UlEZW9Er2TllNqAy GRxBc/sysj2fmOyCWws3ZDkstxCDcs3w6iRL+tmULsOFFTmpOvaI2vQniaaVT4Si N058JXg6yYNtAheVeH1HqFWD7hPIGRqzPPFf/jsC4YX7EWarCV2fTEPwxyReKXIo ztR1aE8kcimuOSj8341PTYNzdAxvEZun3WLe/+LrF+b/DL/ALTE71lmi8t2HSkh7 bTMRFE00nzI49sgZnfG2PcVG71ELisYz7UhhxB0XG718tmfpOc+lUoAK9OrNAgMB AAGjggNUMIIDUDAfBgNVHSMEGDAWgBRMWMsl8EFPUvQoyIFDm6aooOaS5TAdBgNV HQ4EFgQUh9GPGW7kh29TjHeRB1Dfo79VRyAwJQYDVR0RBB4wHIIKZ2l0aHViLmNv bYIOd3d3LmdpdGh1Yi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG AQUFBwMBBggrBgEFBQcDAjBjBgNVHR8EXDBaMCugKaAnhiVodHRwOi8vY3JsMy5k aWdpY2VydC5jb20vZXZjYTEtZzIuY3JsMCugKaAnhiVodHRwOi8vY3JsNC5kaWdp Y2VydC5jb20vZXZjYTEtZzIuY3JsMIIBxAYDVR0gBIIBuzCCAbcwggGzBglghkgB hv1sAgEwggGkMDoGCCsGAQUFBwIBFi5odHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9z c2wtY3BzLXJlcG9zaXRvcnkuaHRtMIIBZAYIKwYBBQUHAgIwggFWHoIBUgBBAG4A eQAgAHUAcwBlACAAbwBmACAAdABoAGkAcwAgAEMAZQByAHQAaQBmAGkAYwBhAHQA ZQAgAGMAbwBuAHMAdABpAHQAdQB0AGUAcwAgAGEAYwBjAGUAcAB0AGEAbgBjAGUA IABvAGYAIAB0AGgAZQAgAEQAaQBnAGkAQwBlAHIAdAAgAEMAUAAvAEMAUABTACAA YQBuAGQAIAB0AGgAZQAgAFIAZQBsAHkAaQBuAGcAIABQAGEAcgB0AHkAIABBAGcA cgBlAGUAbQBlAG4AdAAgAHcAaABpAGMAaAAgAGwAaQBtAGkAdAAgAGwAaQBhAGIA aQBsAGkAdAB5ACAAYQBuAGQAIABhAHIAZQAgAGkAbgBjAG8AcgBwAG8AcgBhAHQA ZQBkACAAaABlAHIAZQBpAG4AIABiAHkAIAByAGUAZgBlAHIAZQBuAGMAZQAuMH0G CCsGAQUFBwEBBHEwbzAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu Y29tMEcGCCsGAQUFBzAChjtodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln aUNlcnRIaWdoQXNzdXJhbmNlRVZDQS0xLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqG SIb3DQEBBQUAA4IBAQBfFW1nwzrVo94WnEUzJtU9yRZ0NMqHSBsUkG31q0eGufW4 4wFFZWjuqRJ1n3Ym7xF8fTjP3fdKGQnxIHKSsE0nuuh/XbQX5DpBJknHdGFoLwY8 xZ9JPI57vgvzLo8+fwHyZp3Vm/o5IYLEQViSo+nlOSUQ8YAVqu6KcsP/e612UiqS +UMBmgdx9KPDDzZy4MJZC2hbfUoXj9A54mJN8cuEOPyw3c3yKOcq/h48KzVguQXi SdJbwfqNIbQ9oJM+YzDjzS62+TCtNSNWzWbwABZCmuQxK0oEOSbTmbhxUF7rND3/ +mx9u8cY//7uAxLWYS5gIZlCbxcf0lkiKSHJB319 -----END CERTIFICATE----- subject=/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1 --- No client certificate CA names sent --- SSL handshake has read 4139 bytes and written 446 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session:     Protocol  : TLSv1.2     Cipher    : ECDHE-RSA-AES128-GCM-SHA256     Session-ID: 59D2883BBCE8E81E63E5551FAE7D1ACC00C49A9473C1618237BBBB0DD9016B8D     Session-ID-ctx:      Master-Key: B6D2763FF29E77C67AD83296946A4D44CDBA4F37ED6F20BC27602F1B1A2D137FACDEAC862C11279C01095594F9776F79     Key-Arg   : None     PSK identity: None     PSK identity hint: None     SRP username: None     Start Time: 1393392673     Timeout   : 300 (sec)     Verify return code: 0 (ok) 

---

Shouldn't requests/openssl already know where to find valid certs?

No. OpenSSL trusts nothing by default. Its a polar opposite of a browser's model, where nearly everything is trusted by default.

---

 $ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`  ...  >>> r = requests.get('https://github.com/', verify='path to cacert.pem file') 

Why would you trust hundreds of CAs and subordinate CAs (re: cacert.pem) when you know the one CA that is certifying the public key for the site? Trust the one required root and nothing more: DigiCert High Assurance EV CA-1.

Trusting everything - as in the browser's model - is what allowed Comodo Hacker to spoof certificates for Gmail, Hotmail, Yahoo, etc when the Diginotar root was compromised.