Can I obtain TLS secrets from an HTTP client to decrypt my own HTTPS conversation?

Question

I am building a website recorder that acts like a proxy, in order to test web scrapers on an ongoing basis. It is split into three Docker containers, all on GNU/Linux: (1) a proxy, (2) an API and request queue, and (3) a simple web app.

It works fine for HTTP sites: I click a button in the web app, this makes a request to the API container, and that adds something to an internal request queue, which then requests the site via the proxy. The proxy records the site as it passes through.

However, I'd forgotten that one cannot record HTTPS site traffic, and now I've come to implement this, I've found that proxies just use the CONNECT verb, and then act as a data exchanger between the client and the target. I believe I cannot replay the same data chunks as part of the encryption uses a randomised, throwaway, symmetric key (however I have a script suitable for testing this, so I will do so just for the educational value!).

So, I was wondering if my fetching client could give up enough secrets for the proxy system to decode the byte-stream? I am using Wget to do the fetch, which I guess would be using OpenSSL. It does not need to be Wget though: if I were using a PHP script with file_get_contents with a stream context, can I ask the openssl module for the decryption keys?

(To be fair, I will probably not solve the problem in this fashion even if it is possible, I just thought it would be really interesting to learn a bit more about TLS. In practice, I will record a "null" entry against all secure websites in the proxy, and require the requesting service to notify the proxy of header/body data via an API call, so it can be later played back. They will of course have plaintext copies of these items).

Answer 1

Yes, I think you have a couple of options here.

HTTPS is specifically designed to thwart "Man in the Middle" attacks and eavesdroppers, which is essentially what you are trying to achieve. You can break some of its assumptions though, and defeat it.

At the start of the SSL connection, 1. the remote server presents its public key and its cert, 2. the client verifies the cert and 3. sends a session key encrypted with the server's public key. For more details, see e.g. An overview of the SSL or TLS handshake

You have two possible ways to circumvent this protection, in the scenario you describe:

1. Rewrite the TLS data, replacing the server's cert and key with your own

Since you control the communications channel, you could substitute the server's public key & cert with one that you control, at step (1). If you then ask the client to skip step (2) using the --no-check-certificate argument to wget, you can then have full access to the encrypted data.

This is how the Fiddler debugging proxy allows access to HTTPS traffic, see https://www.fiddlerbook.com/fiddler/help/httpsdecryption.asp

2. Retrieve the session key from the client application

Since the client app knows the session key, if you could extract it, you could then decrypt the stream. I think this is what you had in mind in the question.

wget itself has no options to allow logging the session key (see "HTTPS (SSL/TLS) Options"), but it does look like its TLS library, "GnuTLS" has a debugging option that will do what you want, see "Debugging and auditing" in the GnuTLS docs:

SSLKEYLOGFILE When set to a filename, GnuTLS will append to it the session keys in the NSS Key Log format. That format can be read by wireshark and will allow decryption of the session for debugging.

Try setting the SSLKEYLOGFILE environment variable to a filename, and see if wget will then log your TLS session keys to that file? You might need to recompile wget with a debug build of GnuTLS. I haven't tried this myself.