Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to get request headers in Selenium

https://www.sahibinden.com/en

If you open it incognito window and check headers in Fiddler then these are the two main headers you get: enter image description here

When I click the last one and check request headers this is what I get enter image description here

I want to get these headers in Python. Is there any way that I can get these using Selenium? Im a bit clueless here.

like image 614
user3102085 Avatar asked Jun 08 '20 12:06

user3102085


3 Answers

The bottom line is, No, you can't retrieve the request headers using Selenium.


Details

It had been a long time demand from the Selenium users to add the WebDriver methods to read the HTTP status code and headers from a HTTP response. We have discussed about implementing this feature through Selenium at length within the discussion WebDriver lacks HTTP response header and status code methods.

However, Jason Leyba (Selenium contributor) in his comment straightly mentioned:

We will not be adding this feature to the WebDriver API as it falls outside of our current scope (emulating user actions).

Ashley Leyba further added, attempting to make WebDriver the ideal web testing tool will suffer in overall quality as driver.get(url) blocks until the browser has loaded the page and return the response for the final loaded page. So in case of a login redirects, status codes and headers will always end up with a 200 instead of the 302 you're looking for.

Finally, Simon M Stewart (WebDriver creator) in his comment concluded that:

This feature isn't going to happen. The recommended approach is to either extend the HtmlUnitDriver to access the information you require or to make use of an external proxy that exposes this information such as the BrowserMob Proxy

like image 34
undetected Selenium Avatar answered Sep 21 '22 11:09

undetected Selenium


You can use Selenium Wire. It is a Selenium extension which has been developed for this exact purpose.

https://pypi.org/project/selenium-wire/

An example after pip install:

##  Import webdriver from Selenium Wire instead of Selenium
from seleniumwire import webdriver

##  Get the URL
driver = webdriver.Chrome("my/path/to/driver", options=options)
driver.get("https://my.test.url.com")

##  Print request headers
for request in driver.requests:
  print(request.url) # <--------------- Request url
  print(request.headers) # <----------- Request headers
  print(request.response.headers) # <-- Response headers
like image 76
Konemiees Avatar answered Sep 20 '22 11:09

Konemiees


You can run JS command like this;

var req = new XMLHttpRequest()
req.open('GET', document.location, false)
req.send(null)
return req.getAllResponseHeaders()

On Python;

driver.get("https://t.me/codeksiyon")
headers = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', document.location, false);req.send(null);return req.getAllResponseHeaders()")

# type(headers) == str

headers = headers.splitlines()
like image 34
raifpy Avatar answered Sep 18 '22 11:09

raifpy