Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

selenium-wire blocking connection due to proxy config

I'm using selenium-wire and the firefox webdriver to access a website (online game). I am running a python script on my local network and no proxy is needed to access the Internet.

Here is a extract on my code:

#!C:/Python38-32/python.exe
    
from seleniumwire import webdriver  # Import from seleniumwire
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains

# Create a new instance of the Firefox driver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

binary = FirefoxBinary(r'C:\\Program Files\\Mozilla Firefox\\firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary)

# Go to the home page
driver.get('https://fr0.forgeofempires.com/page/')

iframe = driver.find_element_by_tag_name('iframe')
iframe_switched = driver.switch_to.frame(iframe)
    
useridInput = driver.find_element_by_id('login_userid')
useridInput.click();
    
useridInput.send_keys('myuser'); 

login_passwordInput = driver.find_element_by_id('login_password')
login_passwordInput.click();

login_passwordInput.send_keys('mypass'); 

loginButton = driver.find_element_by_id('login_Login')
loginButton.click();

defaultContent_switched = driver.switch_to.default_content()

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "play_now_button"))
)
play_nowButton = driver.find_element_by_id('play_now_button')
play_nowButton.click();

WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.LINK_TEXT, "ServerName"))
)
Button = driver.find_element_by_link_text('ServerName')
Button.click();

Everything works fine until this point. The page is now supposed to load the game at a new URL (https://xxx.forgeofempires.com/game) but it gets stuck. If I hit F5 I get The proxy server is refusing connections. I went to my Firefox proxy settings and noticed that they changed from No proxy or Use system proxy settings to Manual 127.0.0.1. I guess it it Selenium-wire changing that in order to inspect the traffic?

enter image description here

My ultimate goal is to catch XHR responses generated ny the page, that's why I use Selenium-wire.

What could cause the localhost proxy to get blocked when connecting to the website? How can I fix that?

If I replace

from seleniumwire import webdriver  # Import from seleniumwire

by

from selenium import webdriver  # Import from selenium

it works fine, but then I won't be able to catch the XHR responses. I have also tried mitmproxy but didn't managed to get it working.

UPDATE 1:

I've made a fully non-working example which shows that there is something wrong done by Selenium-wire. In the example below, the result page of Google cannot be loaded.

#!C:/Python38-32/python.exe
    
from seleniumwire import webdriver  # Import from seleniumwire
#from selenium import webdriver  # Import from selenium

from selenium.webdriver.common.keys import Keys

# Create a new instance of the Firefox driver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

binary = FirefoxBinary(r'C:\\Program Files\\Mozilla Firefox\\firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary)

driver.maximize_window()

# Go to the home page
driver.get('https://www.google.se/')

useridInput = driver.find_element_by_name('q')
useridInput.click();
    
useridInput.send_keys('test'); 

driver.find_element_by_name("q").send_keys(Keys.ENTER)
like image 765
remyremy Avatar asked Jul 17 '20 07:07

remyremy


People also ask

Can you use proxies with selenium?

The best way to integrate authenticated proxies with Selenium is by using PhantomJS as a headless browser instead of the Chrome WebDriver. However, it is also possible to add a browser extension that does the authentication for Selenium.

Can we handle a proxy using selenium in Java?

We can handle proxy in Selenium in Java with the help of PROXY class.

What is Seleniumwire?

Selenium Wire extends Selenium's Python bindings to give you access to the underlying requests made by the browser. You author your code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses and making changes to them on the fly.


1 Answers

I added this line in the end to make it work:

value = input("SCRIPT ENDED\n")

The reason for that can be found here and below:

Selenium Wire works by transparently configuring the browser to point at Selenium Wire's own proxy server. That proxy server (running on 127.0.0.1:49818 in your example above) is used to capture requests made by the browser whilst Selenium Wire is running. After Selenium Wire ends, it shuts down its proxy server because it thinks it's done. However, if the browser is left open (which seems to be the case in your example above), the browser will still be pointing at the proxy server. Trying to use the browser will not work, because the proxy server has now gone away with the shutdown of Selenium Wire.

If you wish to manually interact with the browser, you'll need to keep Selenium Wire running by ensuring that your program does not end. You could potentially use a time.sleep() for that, or perhaps some other mechanism such as input().

like image 161
remyremy Avatar answered Sep 28 '22 19:09

remyremy