How to run selenium chrome nodes using proxy?

Question

I'm using Docker Selenium images to run browser nodes, repo is available here https://github.com/SeleniumHQ/docker-selenium. There is no documentation on how config.json can be used to provide proxy values.

I'm using Selenium version 2.44.0.

In my infrastructure, there are certain assets that are sourced from a location which needs proxy configuration on browser to access them. I'm trying to setup proxy on a chrome node. According to this documentation here, proxy can be set like following:

java -jar selenium-2.44.0.jar -Dhttp.proxyHost=192.168.2.10 -Dhttp.proxyPort=80

My proxy does not require, usename and password hence I have ignored those values. What is not clearly mentioned on SeleniumHQ documentation is, whether it needs proxy configuration on both hub or nodes or just the nodes. I've tried different combinations but haven't worked for me.

Actual commands i'm running are:

For Hub:

java -jar /opt/selenium/selenium-server-standalone.jar -role hub -Dhttp.proxyHost=192.168.2.10 -Dhttp.proxyPort=80 -hubConfig /opt/selenium/hubconfig.json

When I run command above, I can see -D* values being displayed on console config.

For node:

xvfb-run --server-args=":99.0 -screen 0 1360x1020x24 -ac +extension RANDR" java -jar /opt/selenium/selenium-server-standalone.jar -Dhttp.proxyHost=192.168.2.10 -Dhttp.proxyPort=80  -role node -hub http://$HUB_PORT_4444_TCP_ADDR:$HUB_PORT_4444_TCP_PORT/grid/register -nodeConfig /opt/selenium/config.json

When I run this command I can see the proxy values on console again but I the assets are not loaded by the browser.

Also, on a side note it seems like this can be done on developers side (in java code) but I'm keen to solve it on my (operations) side.

Answer 1

Thanks - here is what we got:

First you need a way to verify your settings made it into the browser.

chrome://net-internals/proxyservice.config#proxy

The actual command line instruction is:

/chromeexec --proxy="http=http://proxyserver:port/;https=http://proxyserver:port/"

Note that the colons will blow up on the bash command line if you don't use double-quotes.

Now if you're sending this from the Webdriver Java code programmatically - you'll need to escape out the double quotes - so the proxy server setting in Java may look like:

org.openqa.selenium.Proxy proxy = new org.openqa.selenium.Proxy();
proxy.setHttpProxy("\"http://proxyserver:port/\"")

Alternatively you can pass this in as an execution parameter.

DesiredCapabilities capabilities = DesiredCapabilities.chrome();
capabilities.setCapability("chrome.switches", Arrays.asList("--proxy \"http=http://proxyserver:port/;https=http://proxyserver:port/\""));
WebDriver driver = new ChromeDriver(capabilities);

Now your origin question was about accessing external resources with the proxy. What we did (similar to your question) was to pass a proxy exception for the site we were hitting so the external resources would go via the proxy.

So then you add an exception for your primary website - assuming the resource is 10.1.10.5 then it looks like:

--proxy-bypass-list=10.1.10.5

Which then we do in code as:

capabilities.setCapability("chrome.switches", Arrays.asList("--proxy=\"http=http://proxyserver:port/;https=http://proxyserver:port/\"" "--proxy-bypass-list=10.1.10.5"));

Note that setting username and password is a bug in Chrome. (Please star it if this holds you up. )

If you need a username and password, then the solution is a PAC file.

The syntax is:

--proxy-pac-url=file:///proxy.pac

The file format looks like:

if (host == "mylocalserver.com")
{
    return 'DIRECT';
} else {
    return return "PROXY wcg2.example.com:8080 ";
}

For the case of usernames and passwords in proxy settings, note the following:

Proxy auto-configuration files do not support hard-coded usernames and passwords. There's good reasoning behind this too, since providing support for hard-coded credentials would open up significant security holes, as anybody would be able to easily view the required credentials to access the proxy.

Rather configure the proxy as a transparent proxy, that way you won't need a username and password. You mention in one of your comments that the proxy server is located outside your LAN, which is why you require authentication. However, most proxies support rules based on the source IP, in which case it's a simple matter of only allowing requests originating from your corporate network.

The original proxy auto-config specification was originally drafted by Netscape in 1996. The original specification is no longer available directly, but you can still access it using The Wayback Machine's archived copy. The specification hasn't changed much, and is still largely the same as it was originally. You'll see the specification is quite simple, and that there is no provision for hard-coded credentials.

To solve this problem - you can use this tool:

https://github.com/sjitech/proxy-login-automator

This tool can create a local proxy and automatically inject user/password to real proxy server. Support PAC script.