Unable to access unsecured https with selenium using browsermob proxy

Question

I'm trying to embed browsermob proxy with my selenium (chrome) framework for UI automated testing in order to intercept responses and other networking.

Description :

Selenium webdriver using browsermob proxy and it works just fine - HTTP and secured HTTPS URL's are OK. When I'm trying to navigate to unsecured HTTPS URL I get this chrome error: ERR_TUNNEL_CONNECTION_FAILED

Here's my python code:

class Browser(object):
    display = None
    browser = None

    def __init__(self, implicitly_wait_seconds=10, is_visible=True, display_size=None, browser_name='chrome'):
        if not is_visible:
            self.display = Display(display_size)
        self.server = Server('/home/erez/Downloads/browsermob-proxy-2.1.4/bin/browsermob-proxy')
        self.server.start()
        self.proxy = self.server.create_proxy()
        self.capabilities = DesiredCapabilities.CHROME
        self.proxy.add_to_capabilities(self.capabilities)
        self.proxy.new_har("test", options={'captureHeaders': True, 'captureContent': True})
        self.start_browser(display_size, implicitly_wait_seconds, browser_name)

    def __enter__(self):
        return self

    def __exit__(self, _type, value, trace):
        self.close()

    def start_browser(self, display_size, implicitly_wait_seconds=10, browser_name='chrome'):
        if browser_name == 'chrome':
            chrome_options = Options()
            # chrome_options.add_argument("--disable-extensions")
            chrome_options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors"])
            chrome_options.add_argument("--ssl-version-max")
            chrome_options.add_argument("--start-maximized")
            chrome_options.add_argument('--proxy-server=%s' % self.proxy.proxy)
            chrome_options.add_argument('--ignore-certificate-errors')
            chrome_options.add_argument('--allow-insecure-localhost')
            chrome_options.add_argument('--ignore-urlfetcher-cert-requests')
            self.browser = webdriver.Chrome(os.path.dirname(os.path.realpath(__file__)) + "/chromedriver",
                                            chrome_options=chrome_options, desired_capabilities=self.capabilities)
            self.browser.implicitly_wait(implicitly_wait_seconds)

Answer 1

I faced the same problem for SSL proxying using BrowserMob Proxy. For this you have to install a certificate in your browser that has been defined in this link

Go to the bottom of the document, and you will see a section called "SSL support". Install the ca-certificate-rsa.cer in your browser and there would be no issue in SSL proxying anymore.

Answer 2

Try use this

self.capabilities['acceptSslCerts'] = True

Answer 3

If installing Browsermobs/test servers certificates won't do the job, like in my case, not the most elegant way, but gets the job done:

You are able to bypass ERR_TUNNEL_CONNECTION_FAILED error by passing a trustAllServers-parameter to the proxy instance, which will disable the certificate verification for upstream servers. Unfortunately, for as far as I know, this functionality has not been implemented in the Browsermobs Python wrapper.

However, you can start the proxy with the parameter via Browsermobs REST API (see section REST API @ https://github.com/lightbody/browsermob-proxy/blob/master/README.md). In case of Python, Requests might be the way to go.

Here's a snippet:

import json
import requests
from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Start the proxy via python
server = Server('/path_to/bin/browsermob-proxy')
server.start()

# Start proxy instance with trustAllServers set to true, store returned port number
r = requests.post('http://localhost:8080/proxy', data = {'trustAllServers':'true'})
port = json.loads(r.text)['port']

# Start Chromedriver, pass on the proxy
chrome_options = Options()
chrome_options.add_argument('--proxy-server=localhost:%s' % port)
driver = webdriver.Chrome('/path_to/selenium/chromedriver', chrome_options=chrome_options)

# Get a site with untrusted cert
driver.get('https://site_with_untrusted_cert')

Also, if you need to access HAR-data, you'll need to do that also trough the REST API:

requests.put('http://localhost:8080/proxy/%s/har' % port, data = {'captureContent':'true'})

r = requests.get('http://localhost:8080/proxy/%s/har' % port)

Of course, since you're disabling safety features, this parameter should be used for limited testing purposes only.

Answer 4

You can also call create_proxy with trustAllServers as argument:

self.proxy = self.server.create_proxy(params={'trustAllServers':'true'})