How to get browser network logs using python selenium

Question

I'm trying to get browser network logs using selenium to debug request/responses. Could you please help me to find out a way.

And I'm using selenium 3.14.0 and latest Chrome browser.

Answer 1

Using python + selenium + firefox

Don't set up a proxy unless you have to- in order to get outbound API requests I used the solution from this answer, but in python: https://stackoverflow.com/a/45859018/14244758

test = driver.execute_script("var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return network;")

for item in test:
  print(item)

You get an array of dicts.

This allows me to see all the network requests made. I'm using it to parse out a parameter from one of the requests so that I can use it to make my own requests against the API.

Answer 2

Using Python and ChromeDriver

To get network logs, you need to install BrowserMobProxy as well along with selenium in python

pip install browsermob-proxy

Then we need to download the browsermobproxy zip from https://bmp.lightbody.net/.

Unzip it to any folder(For e.g. path/to/extracted_folder). This folder contains the browsermob-proxy binary file. We need to mention this path while calling Server() in python code

You need to start browser proxy and configure the proxy in chrome option of chrome driver,

from browsermobproxy import Server
from selenium import webdriver

server = Server("path/to/extracted_folder/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

# Configure the browser proxy in chrome options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))
browser = webdriver.Chrome(chrome_options = chrome_options)

#tag the har(network logs) with a name
proxy.new_har("google")

Then you can navigate to page using selenium

browser.get("http://www.google.co.in")

After navigation, you can get the network logs in json format from the proxy

print(proxy.har) # returns a Network logs (HAR) as JSON 

Also before quitting the driver, stop the proxy server as well at the end,

server.stop()
browser.quit()