Intercepting proxy's certificates generated on-the-fly provoke browser errors

Question

I've written an intercepting proxy in Python 3 which uses a man-in-the-middle "attack" technique to be able to inspect and modify pages coming through it on the fly. Part of the process of "installing" or setting up the proxy involves generating a "root" certificate which is to be installed in the browser and every time a new domain is hit via HTTPS through the proxy, the proxy generates a new site certificate on-the-fly (and caches all certificates generated to disk so it doesn't have to re-generate certificates for domains for which certificates have already been generated) signed by the root certificate and uses the site certificate to communicate with the browser. (And, of course, the proxy forges its own HTTPS connection to the remote server. The proxy also checks the validity of the server certificate if you're curious.)

Well, it works great with the browser surf. (And, this might be relevant -- as of a few versions back, at least, surf didn't check/enforce certificate validity. I can't attest to whether that's the case for more recent versions.) But, Firefox gives a SEC_ERROR_REUSED_ISSUER_AND_SERIAL error on the second (and all later) HTTPS request(s) made through the proxy and Chromium (I haven't tested with Chrome proper) gives NET::ERR_CERT_COMMON_NAME_INVALID on every HTTPS request. These obviously present a major problem when trying to browse through my intercepting proxy.

The SSL library I'm using is pyOpenSSL 0.14 if that makes any difference.

Regarding Firefox's SEC_ERROR_REUSED_ISSUER_AND_SERIAL error, I'm pretty sure I'm not reusing serial numbers. (If anybody wants to check my work, that would be pretty rad: cert.py - note the "crt.set_serial_number(getrandbits(20 * 8))" on line 168.) The root certificate issuer of course doesn't change, but that wouldn't be expected to change, right? I'm not sure what exactly is meant by "issuer" in the error message if not the root certificate issuer.

Also, Firefox's "view certificate" dialog displays completely different serial numbers for different certificates generated by the proxy. (As an example, I've got one generated for www.google.com with a serial number of 00:BF:7D:34:35:15:83:3A:6E:9B:59:49:A8:CC:88:01:BA:BE:23:A7:AD and another generated for www.reddit.com with a serial number of 78:51:04:48:4B:BC:E3:96:47:AC:DA:D4:50:EF:2B:21:88:99:AC:8C .) So, I'm not really sure what Firefox is complaining about exactly.

My proxy reuses the private key (and thus public key/modulus) for all certificates it creates on the fly. I came to suspect this was what Firefox was balking about and tried changing the code to generate a new key pair for every certificate the proxy creates on the fly. That didn't solve the problem in Firefox. I still get the same error message. I have yet to test whether it solves the Chromium issue.

Regarding Chromium's NET::ERR_CERT_COMMON_NAME_INVALID error, the common name for site certificate is just supposed to be the domain, right? I shouldn't be including a port number or anything, right? (Again, if anybody would like to check my work, see cert.py .) If it helps any, my intercepting proxy isn't using any wildcards in the certificate common names or anything. Every certificate generated is for one specific fqdn.

I'm quite certain making this work without making Firefox or Chrome (or Chromium or IE etc) balk is possible. A company I used to work for purchased and set up a man-in-them-middling proxy through which all traffic from within the corporate network to the internet had to pass. The PC administrators at said company installed a self-signed certificate as a certificate authority in every browser on every company-owned computer used by the employees and the result never produced any errors like the ones Firefox and Chromium have been giving me for the certificates my own intercepting proxy software produces. It's possible the PC administrators tweaked some about:config settings in Firefox to make this all work or something, but I kindof doubt it.

To be fair, the proxy used at this company was either network or transport layer, not application layer like mine. But I'd expect the same can be accomplished in an application-layer HTTP(s) proxy.

Edit: I've tried setting the subjectAltName as suggested by brain99. Following is the line I added in the location brain99 suggested:

r.add_extensions([crypto.X509Extension(b"subjectAltName", False, b"DNS:" + cn.encode("UTF-8"))])

I'm still getting SEC_ERROR_REUSED_ISSUER_AND_SERIAL from Firefox (on the second and subsequent HTTPS requests and I'm getting ERR_SSL_SERVER_CERT_BAD_FORMAT from Chromium.

Here are a couple of certificates generated by the proxy:

google.com: https://pastebin.com/YNr4zfZu

stackoverflow.com: https://pastebin.com/veT8sXZ4

Answer 1

I noticed you only set the CN in your X509Req. Both Chrome and Firefox require the subjectAltName extension to be present; see for example this Chrome help page or this Mozilla wiki page discussing CA required or recommended practices. To quote from the Mozilla wiki:

Some CAs mistakenly believe that one primary DNS name should go into the Subject Common Name and all the others into the SAN.

According to the CA/Browser Forum Baseline Requirements:

• BR #9.2.1 (section 7.1.4.2.1 in BR version 1.3), Subject Alternative Name Extension
  • Required/Optional: Required
  • Contents: This extension MUST contain at least one entry. Each entry MUST be either a dNSName containing the Fully-Qualified Domain Name or an iPAddress containing the IP address of a server.

You should be able to do this easily with pyOpenSSL:

if not os.path.exists(path):
    r = crypto.X509Req()
    r.get_subject().CN = cn
    r.add_extensions([crypto.X509Extension("subjectAltName", False, "DNS:" + cn])
    r.set_pubkey(key)
    r.sign(key, "sha1")

If this does not solve the issue, or if it only partially solves it, please post one or two example certificates that exhibit the problem.

---

Aside from this, I also noticed you sign using SHA1. Note that certificates signed with SHA1 have been deprecated in several major browsers, so I would suggest switching to SHA-256.

r.sign(key, "sha256")