Getting "SSL_connect returned=1 errno=0 state=error: certificate verify failed" when connecting to S3

Question

I have been trying to upload a photo to my AWS bucket, but running into the error mentioned in the title. I understand that it most likely has to do with my OpenSSL certificates, but any suggested solution that I have tried has failed thus far.

I am running into this issue with ruby 2.3.1, Rails 4.1.8, aws-sdk-core 2.3.4, and carrierwave 0.11.0 on OSX Yosemite.

I have tried all available found at this similar issue as well, as others (this one being with Windows): https://github.com/aws/aws-sdk-core-ruby/issues/166#issuecomment-111603660

Here are some of my files:

carrierwave.rb

CarrierWave.configure do |config|                     # required
  config.aws_credentials = {
    access_key_id:     Rails.application.secrets.aws_access_key_id, # required
    secret_access_key: Rails.application.secrets.aws_access_key,    # required
    region:            'eu-west-2'                  # optional, defaults to 'us-east-1'
  }

  config.aws_bucket = Rails.application.secrets.aws_bucket                        # required
  config.fog_attributes = { 'Cache-Control' => "max-age=#{365.day.to_i}" } # optional, defaults to {}
end

avatar_uploader.rb

class AvatarUploader < CarrierWave::Uploader::Base

  storage :aws

  def store_dir
    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end
end

EDIT (more info):

stack trace:

    Seahorse::Client::NetworkingError - SSL_connect returned=1 errno=0 state=error: certificate verify failed:
  /Users/stevenharlow/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock'
  /Users/stevenharlow/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:933:in `connect'
  /Users/stevenharlow/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
  /Users/stevenharlow/.rbenv/versions/2.3.1/lib/ruby/2.3.0/net/http.rb:858:in `start'
  /Users/stevenharlow/.rbenv/versions/2.3.1/lib/ruby/2.3.0/delegate.rb:83:in `method_missing'
  aws-sdk-core (2.3.4) lib/seahorse/client/net_http/connection_pool.rb:292:in `start_session'
  aws-sdk-core (2.3.4) lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'
  aws-sdk-core (2.3.4) lib/seahorse/client/net_http/handler.rb:109:in `session'

Solutions tried:

• Aws.use_bundled_cert!
• Download cert and reference manually
• I tried using Fog instead of carrierwave-aws
• Tried reinstalling ruby after upgrading rbenv

Here's the result of

CONNECTED(00000003)
depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./CN=*.s3-us-west-2.amazonaws.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
---

<certificate info>

No client certificate CA names sent
---
SSL handshake has read 2703 bytes and written 456 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: <session-id>
    Session-ID-ctx: 
    Master-Key: <master-key>
    Key-Arg   : None
    Start Time: 1463697130
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

Answer 1

With the investigative help of @RodrigoM and your question update, it all started to make sense. There are actually two distinct problems that contribute to the error you observe:

• Your openssl installation does not have the certificate chain needed to verify the Amazon server in its trusted certs store...
• ...which is the exact situation that should be solved by adding Aws.use_bundled_cert! to an initializer, according to the docs. But in this case it does not work because even though this command instructs the ruby openssl library to add various CA certs to the trusted store from the aws-sdk-core gem's CA bundle file, the file also does not contain the proper CA certificate as it is itself almost 2 years old and outdated. The intermediate CA cert CN=DigiCert Baltimore CA-2 G2 has been published Dec 8, 2015, so no wonder that the CA bundle file does not contain it.

Now, you have two options:

• You can try to install this intermediate CA certificate, probably including the root CA cert (CN=Baltimore CyberTrust Root), to your openssl trusted certs store. This should make the s_client command work. But you might still run into issues using these trusted certs from ruby code. For concrete steps for making it work under ruby on OSX, refer to the Solution section of this SO question.

• Also, since you are using a forked aws-sdk-ruby gem repository anyway, you may as well update the ca-bundle.crt file in your repo by adding the intermediate CA cert yourself (the root CA cert seems to be already present in the bundle). For this you need to do the following:

  • download the intermediate CA cert from the official page of the DigicertCA certificates (you can as well use the direct link above, but to obey security rules precisely you should also check the fingerprints)

  • convert it to the PEM format (it gets downloaded in DER format) and add it to the cert bundle using the following openssl command:

    openssl x509 -in DigiCertBaltimoreCA-2G2.crt -inform DER >> ca-bundle.crt
    

    after running this command, your ca-bundle.crt should contain the intermediate CA certificate at the end of the file.

  • Now simply push this updated bundle file to your repo and the Aws.use_bundled_cert! should start working!

  • If you care, perhaps the best would be also to start a github issue at the aws-sdk-ruby gem so that they update the cert bundle in their repo too...

Answer 2

Your Ruby code, AWS SDK etc are all good. This is not a Ruby or SDK issue. The error message you initially describe, and your later post of the OpenSSL connect log both point to the root cause of the problem: A missing root certificate and/or improperly configured CA cert bundle in the OpenSSL framework. A further clue is that the same code works on production. Its not the code.

The original error message itself points to OpenSSL certificate verification errors as you indicate. The stack traces also show Ruby internal lib errors in 2.3.1/lib/ruby/2.3.0/net/http.rb. This is core network library that leverages the OpenSSL framework.

The openssl s_client connect log more clearly shows the extact error number and message:

depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
verify error:num=20:unable to get local issuer certificate

Until you get a Verify return code: 0 (ok) on the DigiCert Baltimore CA-2 CA with the openssl s-client test, your code will not work.

The DigiCert Baltimore CA-2 CA cert is not present or not properly referenced by your OpenSSL setup on that machine. This is very common problem in broken or incomplete OpenSSL installations. You need to download that certificate, convert to PEM format, save it in a ca-certificate.crt file in your OpenSSL certs folder, then reference that file in your config, or in an environment variable SSL_CERT_FILE.

You can see a good solution for your root issue here

NOTE: To futher confirm this solution, you should run the openssl s_client test on your production server. You should see it verifying the same CA without problem. Check the OpenSSL config and CA cert bundle configuration there to see why there is a difference between your production and development environment.