How to force Apache 2.2 to send the full certificate chain?

Question

We are using Apache 2.2.25 with mod_ssl in the reverse proxy mode using mod_proxy. It has a server certificate we use for testing purposes, issued by GoDaddy. There are 3 certificates in the chain, server cert -> GoDaddy intermediate CA -> GoDaddy Root CA. The intermediate CA (Go Daddy Secure Certificate Authority - G2) is not always found in clients' list of trusted CA.

The SSL connection to the server works well for browsers (at least for some), but not for some other clients. We noticed that our server does not send the full certificate chain, by using the following command: openssl s_client -showcerts -connect SERVER_URL:443, and indeed the command reports the error Verify return code: 21 (unable to verify the first certificate)

We use the SSLCertificateFile directive in each VirtualHost:

SSLCertificateFile certificate.crt

Where the certificate.crt file contains the private key and all the certificates in the chain. We tried to split it into the following:

SSLCertificateFile server.crt
SSLCertificateKeyFile server.key
SSLCertificateChainFile chain.crt

But this didn't change anything.

Thanks for your help!

EDIT
The plot thickens - it seems to be some combination of the certificate and the server.
(testing is done with the SSL Shopper tool)

1. Go Daddy certificate (as above) on Apache 2.2 (RHEL) - does not work
2. same certificate, on IIS7 - works
3. customer's certificate (from Comodo) on Apache 2.2 RHEL - works

Answer 1

You can also use the SSLCACertificatePath directive and put the original .crt files into the directory specified. However, you also have to create hash symlinks to them. This is done with the c_rehash tool, which is part of openssl. For example,

sudo c_rehash /etc/apache2/ssl/certs

However, note that there are two hash algorithms in use. The new one was introduced with openssl 1.0 and it's necessary to re-run c_rehash after upgrading openssl to 1.0 or later. This will create both old-style and new-style symlinks.

If you don't do this, openssl (and therefore apache) won't be able to find the intermediate certificates and so they won't be sent to the client. I spent a frustrating few hours debugging SSL errors after upgrading an Ubuntu server from Lucid to Precise, which had included an upgrade of openssl from 0.9.8 to 1.0.1. I searched but couldn't find any clues on the web about what was going wrong, so had to figure it out myself.

For the record, we weren't getting errors in the browser because it has a bigger set of roots and one of our intermediate certificates must have been in that set. The problem only showed up when using openssl-based command-line programs such as wget, curl and openssl s_client.

Answer 2

In case anyone has similar issues in the future:

In my case where one server didn't send a configured intermediate certificate (but other servers did) - it appeared to be an issue with line endings in the certificate file. Apparently Apache releases prior to 2019 can be rather picky - accepting only certificates with lines ending with new line (NL, char 10, Unix line endings), and silently ignoring certificate with lines ending with carriage return, new line (CR+NL, chars 13 and 10, Windows line endings), or carriage return only (CR, char 13, Mac OS <10).

My intermediate cert had the CR line endings (very odd in this day and age) - converting it to NL endings using a text editor fixed it up, and Apache now sends the intermediate certificate correctly.

Answer 3

I had httpd 2.4 not sending the intermediate certificates defined with SSLCertificateChainFile.

It turned out the certificate files had wrong permission on the file system, and apache simply ignored them. Correct permission mask is 400 :

[root@server ~]# ll /etc/httpd/conf/tls/certs/intermediate_chain.crt 
-rw-r--r-- 1 root root 1728 Mar  2 14:20 /etc/httpd/conf/tls/certs/intermediate_chain.crt 
#   ^  ^ wrong permissions

[root@server conf.d]# chmod 400 /etc/httpd/conf/tls/certs/intermediate_chain.crt 

[root@server ~]# ll /etc/httpd/conf/tls/certs/intermediate_chain.crt 
-rw------- 1 root root 1728 Mar  2 14:20 /etc/httpd/conf/tls/certs/intermediate_chain.crt 
#   ^  ^ correct permissions

Maybe this helps someone