Log HTTP traffic with Webdriver and PhantomJS

Question

How can I log all HTTP requests and responses of a page load over Webdriver with PhantomJS? I am using python and my super simple test script looks like this:

from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('http://www.golem.de')

I already found the capabilities in PhantomJS:

page.onResourceRequested = function (request) {
    console.log('Request ' + JSON.stringify(request, undefined, 4));
};

But I don't know how to stick this together with Selenium Webdriver respectively Ghostdriver. How could I do this?

Answer 1

One way to log all network traffic is to use the wonderful tool strace, logging all network requests (and data) to a file.

strace -s9999 -e trace=network curl http://example.com > /dev/null

Partial output:

sendto(3, "GET / HTTP/1.1\r\nUser-Agent: curl/7.32.0\r\nHost: example.com\r\nAccept: */*\r\n\r\n", 75, MSG_NOSIGNAL, NULL, 0) = 75
recvfrom(3, "HTTP/1.1 200 OK\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=604800\r\nContent-Type: text/html\r\nDate: Sun, 08 Ju...

Answer 2

Another general low level way, but slightly higher level than strace is tcpdump. You could filter to the specific listening port range and destination host of your server app. You can also log the packets for later analysis if needed. Using the -A (ASCII) dump option you can filter for the request to a given page. A simple example for request to localhost on port 80:

tcpdump -i lo -A -nn dst port 80 and dst host `hostname`

I'm sure Wireshark or similar software could do this type of protocol specific filtering too.

Answer 3

As Torben said, the driver.get_log("har") is a solution, and I think it is a best solution to me.

from selenium import webdriver

service_args = ['--ignore-ssl-errors=yes']
driver = webdriver.PhantomJS(service_args=service_args)
driver.get('https://www.google.com/')
screenshot = driver.get_screenshot_as_png()
imgname = "google.png"
save_img = open(imgname, 'a')
save_img.write(screenshot)
save_img.close()
print driver.get_log('har')
driver.quit()

For another solutions, we could refer to: 1. browsermob-proxy, 2. Or using webdriver/firebug to catch the network (seems the netexport cannot been verified by Firefox, and maybe we could use the firebug itself, for the newest firebug, it has the feature to export the har) 3. Same as 2, another solution is here: How to capture all requests made by page in webdriver? Is there any alternative to Browsermob?