Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to run selenium web driver behind a proxy server which needs authentication in python

At present this is my code, but webDriver is showing a pop-up to enter proxy credentials and I don't want this annoying situation, This is not the first time the same question appeared in stackoverflow, but no one replied with a proper answer.

I tried google to find a solution for this problem. I came to know the solution in java, but i dont know how we do it in python.

    PROXY_HOST = "65.49.1.59"
    PROXY_PORT = 60099
    fp = webdriver.FirefoxProfile()
    # Direct = 0, Manual = 1, PAC = 2, AUTODETECT = 4, SYSTEM = 5

    print " im in parse_details"

    fp.set_preference("network.proxy.type", 1)
    fp.set_preference('network.http.phishy-userpass-length', 255)

    fp.set_preference("network.proxy.http", PROXY_HOST)
    fp.set_preference("network.proxy.http_port", PROXY_PORT)
    fp.set_preference("network.proxy.ftp", PROXY_HOST)
    fp.set_preference("network.proxy.ftp_port", PROXY_PORT)
    fp.set_preference("network.proxy.ssl", PROXY_HOST)
    fp.set_preference("network.proxy.ssl_port", PROXY_PORT)
    #fp.set_preference("network.proxy.user_name", 'someusername')
    #fp.set_preference("network.proxy.password", 'somepassword')
    fp.set_preference("network.proxy.no_proxies_on", "") # set this value as desired

    self.driver = webdriver.Firefox(firefox_profile=fp)
    self.driver.get("http://www.whatismyip.com/")

These below statements are guessed by me, and I am not sure whether their syntax is correct or not, even i tried to find out in selenium documentation, but no help. Would you guys throw some light on this.

    #fp.set_preference("network.proxy.user_name", 'someusername')
    #fp.set_preference("network.proxy.password", 'somepassword')

p.s. The same question asked here Selenium using Python: enter/provide http proxy password for firefox

like image 667
USHA Avatar asked Oct 11 '12 20:10

USHA


People also ask

Which of the following code is used in Selenium to configure the use of proxy?

Following piece of code used to set proxy in Selenium. ChromeOptions option = new ChromeOptions(); Proxy proxy = new Proxy(); proxy. setHttpProxy("localhost:5555"); option. setCapability(CapabilityType.

Can you use Selenium without opening browser Python?

We can perform Selenium testing without a browser. This is achieved by triggering the execution in a headless mode. The headless execution can decrease the utilization of key resources and is being adopted widely.


1 Answers

Selenium can't handle with basic authentication neither it works well with popups. But this problem is resolvable. As a solution that worked to me (I found it here) is to add a browser extension that does the authentication for Selenium. It's quite simple. Here is how it works for Chrome and Python:

  1. Create a zip file proxy.zip containing two files:

background.js

var config = {
    mode: "fixed_servers",
    rules: {
      singleProxy: {
        scheme: "http",
        host: "YOU_PROXY_ADDRESS",
        port: parseInt(YOUR_PROXY_PORT)
      },
      bypassList: ["foobar.com"]
    }
  };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "YOUR_PROXY_USERNAME",
            password: "YOUR_PROXY_PASSWORD"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
        callbackFn,
        {urls: ["<all_urls>"]},
        ['blocking']
);

Don't forget to replace YOUR_PROXY_* to your settings.

manifest.json

{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
  1. Add the created proxy.zip as an extension

Python Code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
chrome_options.add_extension("proxy.zip")

driver = webdriver.Chrome(executable_path='chromedriver.exe', chrome_options=chrome_options)
driver.get("http://google.com")
driver.close()

That's it. For me that worked like a charm. If you need to create proxy.zip dynamically or need PHP example then go to the original post

like image 121
Mike Avatar answered Sep 16 '22 14:09

Mike