Heroku Rails Net::HTTP: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

Question

I have a Rails app running on a Heroku server and I'm having trouble communicating with an external server using Net::HTTP over HTTPS. The error I'm receiving whenever I attempt to POST to an external proprietary API over HTTPS is:

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

I've done quite a few hours of Googling around for an answer to the issue I'm encountering, but to no avail. Here's my environment:

• Heroku Dyno running Cedar 14 (was running Cedar 10, upgraded to Cedar 14 to see if it would affect the issue - no joy)
• Rails 3.2.14rc2
• Ruby 2.1.2

"Fixes" I have tried:

• Running the certified gem (which has actually seemed to help out with communication via Omniauth and Google's API)
• Monkey-patch removing SSLv2, SSLv3 from the list of verification methods. When I attempt this in the console on Heroku's server, it appears to work well enough for GET methods, but appears ignore the adjustments completely when attempting to Run Net::HTTP::Post.new instead of Net::HTTP::Get.new
• Manually overriding the certificate file location. I've specified it to use the authority file of the operating system (which on the Cedar-14 stack is up-to-date), but still no dice.
• I have also tried manually specifying in on the Net::HTTP object to use ssl_version="TLSv1_2" with no luck (it even keeps reporting the same SSLv3 related error)

The communication appears to be working just fine when run locally in development (I've used the RVM override method suggested here), but the moment I try things on the Heroku server, I'm out of luck.

UPDATE: It appears this is not working locally either even with the RVM update.

Last, but not least, here is an abstracted variant of the code I'm running:

uri = URI.parse("https://api.mysite.com/api/v1")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = (uri.scheme == "https")
request = Net::HTTP::Post.new(uri.path, {'Content-Type' =>'application/json'})
request_body = "{\"post_body\": \"data1\"}"
response = http.request(request)

Anyone have any advice on what else I should be looking at?

Answer 1

Heroku can't verify your server's certificate is validly signed by a CA root it recognizes. This can be because either:

1. Your cert isn't signed by a CA or intermediate (ie, self-signed)
2. Your cert is signed by a CA that Heroku doesn't know about (unlikely)
3. The API server isn't providing the correct intermediate certs to help Heroku connect it to a valid CA root. (likely)

Try openssl s_client -showcerts -connect your-api-host.com:443 from your shell. You should see something like:

depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = www.coffeepowered.net
verify return:1

You're specifically looking to make sure that all certs in the chain return verify return: 1. If this works from your shell, then your machine likely has root certs installed that your Heroku instance doesn't.

Without knowing exactly what certs your API server is returning, it's hard to answer this definitively, but you probably need to be serving an intermediate cert bundle along with the SSL cert itself. This intermediate cert bundle will be provided by your SSL certificate signer, and can be provided in Apache via SSLCertificateChainFile, or in nginx by concatening the intermediates with your cert (per this documentation).

If you can't alter the configuration of the API server, then your "Manually overriding the certificate file location" solution is probably very close to correct (it's the same thing as the server providing the intermediate cert, except the client does it), but you are likely not providing the correct certificate chain bundle for your API server's certificates. Make sure that you have the correct intermediate certificate chain provided to OpenSSL, and it should work as desired.