Installing Browsermob proxy for getting HTTP codes

Question

I might seem naive while asking this. I need to retrieve various HTTP codes, for few of our test scenarios. Now I know that Selenium cannot do it alone and we have to use a third party resource- may be a proxy like Browsermob.

Has any one already worked on this. I have gone through various tutorials -like one from here but I can't seem to be able to work around using this.

I installed Browsermob using the pip method and so I am giving the path as

  from browsermobproxy import Server
  server = Server('/usr/local/lib/python3.4/dist-packages/browsermobproxy')

However, I am receiving the exception

  Exception: Browsermob-Proxy binary couldn't be found in path provided: /usr/local/lib/python3.4/dist-packages/browsermobproxy 

My question is - is there anything- any file that I might be missing here? Do I need to download any other files or my path is incorrect?

Answer 1

I was a bit confused about this when starting too, but I'm assuming you installed the browsermobproxy package using pip. You also need to actually download the browsermob-proxy binary (available here: https://bmp.lightbody.net/), then point to it from within your python script.

I was using a virtual environment so I put it here:

/Users/username/Envs/ScrapeProj/bin/browsermob-proxy-2.1.2/bin/browsermob-proxy

If you get the error message:

Error: JAVA_HOME is not defined correctly.

Run these commands in the Terminal:

export JAVA_HOME=$(/usr/libexec/java_home)
export PATH=$JAVA_HOME/jre/bin:$PATH

Answer 2

Not sure if you understand browsermob-proxy-py , it's a client implementation instead of server. so you need to install the server browsermob-proxy first(https://github.com/lightbody/browsermob-proxy) and run the proxy with default port.