Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using a http proxy with headless firefox in Selenium webdriver in Python

I'm using Firefox headless like this:

from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium import webdriver
import os
import sys

# Set the MOZ_HEADLESS environment variable which casues Firefox to
# start in headless mode.
os.environ['MOZ_HEADLESS'] = '1'

# Select your Firefox binary.
binary = FirefoxBinary('/usr/bin/firefox', log_file=sys.stdout)

# Start selenium with the configured binary.
driver = webdriver.Firefox(firefox_binary=binary)

But now I want to add a http proxy that requires a user/password. After searching around, I tried the following:

from selenium.webdriver.common.proxy import Proxy, ProxyType

myProxy = "xx.xx.xx.xx:80"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(firefox_binary=binary, proxy=proxy)

I also tried

profile = webdriver.FirefoxProfile() 
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", "xx.xx.xx.xx")
profile.set_preference("network.proxy.http_port", 80)
profile.update_preferences() 
driver=webdriver.Firefox(firefox_binary=binary,firefox_profile=profile)

Finally, I tried adding "socksUsername" and "socksPassword" with the creds to the proxy, more out of desperation than any real hope.

Needless to say none of these works, and testing shows requests are still using my usual IP, not the proxy.

Also system-wide proxy is not an option in this case.

Where should the http proxy credentials live? How can I use a proxy with headless firefox?

Testing

driver.get("https://www.ipinfo.io");
driver.find_element_by_xpath('//h4/following-sibling::p').text
like image 420
fpghost Avatar asked Dec 31 '17 19:12

fpghost


People also ask

How do I make Firefox headless in selenium?

We can make firefox headless programmatically in Selenium. This can be done with the help of the FirefoxOptions class. We shall then create an object option for that class. We shall set the parameter options.

How can we instantiate Firefox browser in headless mode?

You need simply to add the "--headless" argument to the FirefoxOptions object. * However, the current official version of Mozilla Firefox is 56. You need to install the beta 57 version in order to be able to run your tests in headless mode.

How do I run Firefox in selenium Python?

To make Firefox work with Python selenium, you need to install the geckodriver. The geckodriver driver will start the real firefox browser and supports Javascript. Take a look at the selenium firefox code. First import the webdriver, then make it start firefox.


1 Answers

If your proxy requires a username and a password, you have to write it like that :

from selenium.webdriver.common.proxy import Proxy, ProxyType

myProxy = "username:password@proxyDomain:proxyPort"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })

driver = webdriver.Firefox(firefox_binary=binary, proxy=proxy)
like image 82
Reupiey Avatar answered Sep 21 '22 04:09

Reupiey