How to force older debian to forget about DST Root CA X3 Expiration and use ISRG Root X1 - SSL certificate problem: certificate has expired

Question

This relates to DST Root CA X3 Expiration (September 2021)
When searching online for a fix to apply on an older server (Debian 8 in my case) that does call to sites encrypted with letsencrypt with curl, they now seem to fail with the following message:

Example:

curl -fsSL https://deb.nodesource.com/setup_14.x | bash -

Fails silently, then trying it manually and removing the silent flag and bash pipe like this:

curl -L https://deb.nodesource.com/setup_14.x

curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Trying the following commands doesn't solve the issue:

apt update
apt install -y ca-certificates openssl
update-ca-certificates

What can I do!? (answering my own question) ⬇️

Answer 1

disclaimer; I'm no security expert (I know things, but you do you). Make sure you understand what you do before applying whatever fix shared here

Fix by upgrading your instances

Upgrade your instances. This problem won't happen on debian 9 or higher.

In the following example, I had this problem on a ruby:2.4.1 docker image which is based on Debian 8 (could be considered old). Upgrading to more recent docker image fixes this issue. Uprading to a more recent Debian version should also fix the issue.

I confirmed it does not happen when using ruby:2.7.0 docker image based on Debian 11 as shown here:

docker run --rm -it ruby:2.7.4 bash -c "cat /etc/issue"
Debian GNU/Linux 11 \n \l

Fix for Debian 8 by commenting DST_Root_CA_X3.crt from /etc/ca-certificates.conf

Even if ISRG Root X1 is in place, if DST Root CA X3 is still present and in use, its verification seems to happen first so we can get rid of it by doing this:

1. install ca-certificates package
2. comment /mozilla/DST_Root_CA_X3.crt from /etc/ca-certificates.conf
3. make sure /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt is there (it should be)
4. update ca-certificates with update-ca-certificates

Example directly on your instance

cat /etc/issue
Debian GNU/Linux 8 \n \l

sudo apt install -y ca-certificates
sudo sed -i '/^mozilla\/DST_Root_CA_X3.crt$/ s/^/!/' /etc/ca-certificates.conf
sudo update-ca-certificates

Example Dockerfile:

FROM ruby:2.4.1 # uses debian 8

RUN apt update -qq \
    && apt install -y ca-certificates \
    && sed -i '/^mozilla\/DST_Root_CA_X3.crt$/ s/^/!/' /etc/ca-certificates.conf \
    && update-ca-certificates \
    && rm -rf /var/lib/apt/lists/*

Fix using dpkg-reconfigure ca-certificates

As stated in the comments, you can also fix this interactively using the following command on the instance (requires ca-certificates package installed):

dpkg-reconfigure ca-certificates

Then disable mozilla/DST_Root_CA_X3.crt from the list.

Conclusion

If you'd like to learn more, you should read Scott Helme's post: Let's Encrypt's Root Certificate is expiring!

You can now curl letsencrypt sites safely with these safety glasses: 🥽

Answer 2

It doesn't get clear how "old" your Debian is.

Having the latest Linux OS will not cause this issue. Consider before everything upgrading your distro to the latest version!

The following worked on my older Debian and Ubuntu systems.

To clarify the issue is with CA Root X3 certificate part of the "cacert-bundle". As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html as part of the bundle https://curl.se/ca/cacert.pem.

The expired certificate is:

Certificate:
    Data:
    Version: 3 (0x2)
    Serial Number:
    44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
    Not Before: Sep 30 21:12:19 2000 GMT
    Not After : Sep 30 14:01:15 2021 GMT
    Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
    Subject Public Key Info:
    Public Key Algorithm: rsaEncryption
    Public-Key: (2048 bit)

Which is used to verify peer in curl calls to websites using Let's Encrypt issued certificates.

Find which CA Certificates bundle is being used by CURL:

strace curl https://www.google.com |& grep open 

Result:

root@debian:/tmp# strace curl https://www.google.com |& grep open
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/x86_64-linux-gnu/libcurl.so.4", O_RDONLY|O_CLOEXEC) = 3
...
...
open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
open("/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4

Near the end you should see the certificate bundle. In this case: /etc/ssl/certs/ca-certificates.crt

An option to fix this issue is to replace that CA certificate bundle, but that may not be a global solution for your OS and other affected apps and packages (E.g.: php libcurl).

Now to dissect the Root CA bundle certificate, use:

openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs -text -noout | grep "Not After"

NOTE: The cert location (/etc/ssl/certs/ca-certificates.crt) in the command is from the "strace" step above.

This will return long list with expiration dates for all Root CA certificates in the bundle.

root@L36630:/tmp# openssl crl2pkcs7 -nocrl -certfile cacert.pem | openssl pkcs7 -print_certs -text -noout | grep "Not After"
        Not After : Sep 30 04:20:49 2023 GMT
        Not After : Sep 30 14:01:15 2021 GMT
        Not After : Sep 22 11:22:02 2030 GMT
        Not After : Sep 15 16:00:00 2025 GMT
        ...
        ...

In this case, our expired certificate is "Not After : Sep 30 14:01:15 2021 GMT".

You can remove the | grep "Not After" to see the full decrypted list.

This the step-by-step method to troubleshoot the cacert bundle. If this happens in the future you know what to do.

Now let's get to fixing the issue.

FIX

Let’s Encrypt originally used the “DST Root CA X3” CA Root certificate. Let's encrypt now uses “ISRG Root X1” and “ISRG Root X2” as Root CA’s and “Let’s Encrypt R3” as an intermediate certificate.

To fix this issue, you need to add the 2 new Root CAs to your server or device:

• ISRG Root X1 (Or ISRG Root X1 DER Format)
• ISRG Root X2 (Or ISRG Root X2 DER Format)

Intermediate Certificate (PEM format):

• Let’s Encrypt R3 (Or Let’s Encrypt R3 DER Format)

Install Root CA Certificate on Linux:

sudo mkdir /usr/local/share/ca-certificates/extra

sudo cp root.cert.pem /usr/local/share/ca-certificates/extra/root.cert.crt

sudo update-ca-certificates

NODEJS

Node.js 7.3.0 (and the LTS versions 6.10.0 and 4.8.0) added NODE_EXTRA_CA_CERTS environment variable for you to pass the CA certificate file.

$ export NODE_EXTRA_CA_CERTS=[custom Root CA certificate file path]

Alternatively you can blacklist/remove the DST certificate from the CA cert bundle for your OS.

DEBIAN/UBUNTU

You can use the dpkg to reconfigure in Ubuntu/Debian.

sudo dpkg-reconfigure ca-certificates

1. You will see terminal window with "ca-certificates configuration" (ca-certificates configuration image 1). Select "yes" to trust new certificates from certificate authorities.
2. Press "OK"
3. The window will close and in the next step you will see the actual list with CA certs (ca-certificates configuration image 1). Find the cert by name "DST_Root_CA_X3.crt", uncheck it and press "OK".
4. Now you will see "Updating certificates in /etc/ssl/certs...". Wait until the process finishes.

Next step retest your curl call from the terminal. You should no longer see the CURL error:

root@debian:/tmp# curl https://example.com
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Another option (Ubuntu/Debian):

The list of CAs is stored in the file /etc/ca-certificates.conf. You can edit this file manually and run:

sudo update-ca-certificates

To automate it:

# Make sure the ca-certificates.conf location is correct 
sed '/DST_Root_CA_X3.crt/d' /etc/ca-certificates.conf > /tmp/cacerts.conf && mv /tmp/cacerts.conf /etc/ca-certificates.conf
update-ca-certificates

RHEL/CENTOS

In RedHat add that cert to the ca-trust blacklist: /etc/pki/ca-trust/source/blacklist

1. Create the dstrootca.pem file:

DST Root CA X3
==============
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/MSQwIgYDVQQK
ExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMTDkRTVCBSb290IENBIFgzMB4X
DTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVowPzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1
cmUgVHJ1c3QgQ28uMRcwFQYDVQQDEw5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmT
rE4Orz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEqOLl5CjH9
UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9bxiqKqy69cK3FCxolkHRy
xXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40d
utolucbY38EVAjqr2m7xPi71XAicPNaDaeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0T
AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQ
MA0GCSqGSIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69ikug
dB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXrAvHRAosZy5Q6XkjE
GB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZzR8srzJmwN0jP41ZL9c8PDHIyh8bw
RLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubS
fZGL+T0yjWW06XyxV3bqxbYoOb8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

2. Add the file to your blacklist location: /etc/pki/ca-trust/source/blacklist

3. Run sudo update-ca-trust

Source: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

Let's Encrypt formal article: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/