How can you test that an SSL client library is properly verifying the certificate of the server to which it connects?

Question

I want to ensure that client libraries (currently in Python, Ruby, PHP, Java, and .NET) are configured correctly and failing appropriately when SSL certificates are invalid. Shmatikov's paper, The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software, reveals how confusing SSL validation is so I want to thoroughly test the possible failures.

Based on research a certificate is invalid if:

• It is used before its activation date
• It is used after its expiry date
• It has been revoked
• Certificate hostnames don't match the site hostname
• Certificate chain does not contain a trusted certificate authority

Ideally, I think I would have one test case for each of the invalid cases. To that end I am currently testing an HTTP site accessed over HTTPS, which leads to a failure that I can verify in a test like so:

self.assertRaises(SSLHandshakeError, lambda: api.call_to_unmatched_hostname())

This is incomplete (only covering one case) and potentially wrong, so...

How can you test that non-browser software properly validates SSL certificates?

Answer 1

First off, you'll need a collection of SSL certificates, where each has just one thing wrong with it. You can generate these using the openssl command line tool. Of course, you can't sign them with a trusted root CA. You will need to use your own CA. To make this validate correctly, you'll need to install your CA certificate in the client libraries. You can do this in Java, for example, using the control panel.

Once you have the certificates, you can use the "openssl s_server" tool to serve an SSL socket using each one. I suggest you put one certificate on each port.

You now have to use the client library to connect to a port, and verify that you get the correct error message.

I know that Python by default does no certificate validation (look at the manual for httplib.HTTPSConnection). However, m2crypto does do validation. Java by default does do validation. I don't know about other languages.

Some other cases you could test: 1) Wildcard host names. 2) Certificate chaining. I know there was a bug in old browsers where if you had a certificate A signed by the root, A could then sign B, and B would appear valid. SSL is supposed to stop this by having flags on certificates, and A would not have the "can sign" flag. However, this was not verified in some old browsers.

Good luck! I'd be interested to hear how you get on.

Paul

Answer 2

• Certificate hostnames don't match the site hostname

  This is probably the easiest to check, and failure (to fail) there is certainly a good indication that something is wrong. Most certificates for well-known services only use host names for their identity, not IP addresses. If, instead of asking for https://www.google.com/, you ask for https://173.194.67.99/ (for example) and it works, there's something wrong.

For the other ones, you may want to generate your own test CA.

• Certificate chain does not contain a trusted certificate authority

  You can generate a test certificate using your test CA (or a self-signed certificate), but let the default system CA list be used for the verification. Your test client should fail to verify that certificate.

• It is used before its activation date, It is used after its expiry date

  You can generate test certificates using your test CA, with notBefore/notAfter dates that make the current date invalid. Then, use your test CA as a trusted CA for the verification: your test client should fail to validate the certificate because of the dates.

• It has been revoked

  This one is probably the hardest to set up, depending on how revocation is published. Again, generate some test certificates that you've revoked immediately, using your own test CA. Some tools expect to be configured with a set of CRL files next to the set of trusted CAs. This requires some setup for the test itself, but very little online setup: this is probably the easiest. You can also set up a local online revocation repository, e.g. using CRL distribution points or OCSP.

PKI testing can be more complex than that more generally. A full test suite would require a fairly good understanding of the specifications (RFC 5280). Indeed, you may need to check the dates for all intermediate certificates, as well as various attributes for each certificate in the chain (e.g. key usage, basic constraints, ...).

In general, client libraries separate the verification process into two operations: verifying that the certificate is trusted (the PKI part) and verifying that it was issued to the entity you want to connect to (the host name verification part). This is certainly due to the fact these are specified in different documents (RFC 3280/5280 and RFC 2818/6125, respectively).

From a practical point of view, the first two points to check when using an SSL library are:

• What happens when you connect to a known host, but with a different identifier for which the certificate isn't valid (such as its IP address instead of the host)?
• What happens when you connect to a certificate that you know cannot be verified by any default set of trusted anchors (for example, a self-signed certificate or from your own CA).

Failure to connect/verify should happen in both cases. If it all works, short of implementing a full PKI test suite (which require a certain expertise), it's often the case that you need to check the documentation of that SSL library to see how these verifications can be turned on.

Bugs aside, a fair number of problems mentioned in this paper are due to the fact that some library implementations have made the assumption that it was up to their users to know what they were doing, whereas most of their users seem to have made the assumption that the library was doing the right thing by default. (In fact, even when the library is doing the right thing by default, there is certainly no shortage of programmers who just want to get rid of the error message, even if it makes their application insecure.) I would seem fair to say that making sure the verification features are turned on would be sufficient in most cases.

As for the status of a few existing implementations:

• Python: there was a change between Python 2.x and Python 3.x. The ssl module of Python 3.2 has a match_hostname method that Python 2.7 doesn't have. urllib.request.urlopen in Python 3.2 also has an option to configure CA files, which its Python 2.7 equivalent doesn't have. (This being said, if it's not set, verification won't occur. I'm not sure about the host name verification.)

• Java: verification is turned on by default for both PKI and host name for HttpsUrlConnection, but not for the host name when using SSLSocket directly, unless you're using Java 7 and you've configure its SSLParameters using setEndpointIdentificationAlgorithm("HTTPS") (for example).

• PHP: as far as I'm aware, fopen("https://.../") won't perform any verification at all.