Set CA bundle for requests going through HTTPS tunnel

Question

I'm trying to send an HTTPS request through an HTTPS tunnel. That is, my proxy expects HTTPS for the CONNECT. It also expects a client certificate.

I'm using Requests' proxy features.

import requests

url = "https://some.external.com/endpoint"
with requests.Session() as session:
    response = session.get(
        url,
        proxies={"https": "https://proxy.host:4443"},
        # client certificates expected by proxy
        cert=(cert_path, key_path),
        verify="/home/savior/proxy-ca-bundle.pem",
    )
    with response:
        ...

This works, but with some limitations:

1. I can only set client certificates for the TLS connection with the proxy, not for the external endpoint.
2. The proxy-ca-bundle.pem only verifies the server certificates in the TLS connection with the proxy. The server certificates from the external endpoint are seemingly ignored.

Is there any way to use requests to address these two issues? I'd like to set a different set of CAs for the external endpoint.

I also tried using http.client and HTTPSConnection.set_tunnel but, as far as I can tell, its tunnel is done through HTTP and I need HTTPS.

Answer 1

Looking at the source code, it doesn't seem like requests currently supports this "TLS in TLS", ie. providing two sets of clients/CA bundles for a proxied requests.

We can use PycURL which simply wraps libcurl

from io import BytesIO

import pycurl    

url = "https://some.external.com/endpoint"
buffer = BytesIO()

curl = pycurl.Curl()
curl.setopt(curl.URL, url)
curl.setopt(curl.WRITEDATA, buffer)
# proxy settings
curl.setopt(curl.HTTPPROXYTUNNEL, 1)
curl.setopt(curl.PROXY, "https://proxy.host")
curl.setopt(curl.PROXYPORT, 4443)
curl.setopt(curl.PROXY_SSLCERT, cert_path)
curl.setopt(curl.PROXY_SSLKEY, key_path)
curl.setopt(curl.PROXY_CAINFO, "/home/savior/proxy-ca-bundle.pem")
# endpoint verification
curl.setopt(curl.CAINFO, "/home/savior/external-ca-bundle.pem")

try:
    curl.perform()
except pycurl.error:
    pass # log or re-raise
else:
    status_code = curl.getinfo(curl.RESPONSE_CODE)

PycURL will use the PROXY_ settings to establish a TLS connection to the proxy, send it an HTTP CONNECT request. Then it'll establish a new TLS session through the proxy connection to the external endpoint and use the CAINFO bundle to verify those server certificates.