Export all http requests on a specific page to txt/csv

Question

I use SIEGE to test my web server performance. For a more realistic test the best way to go would be to have SIEGE hit the web page (website.com/our-company) and all static assets (.css, .js, .png, .jpg). Everything that you see on the firefox / chrome debbuing tools, except of course from resources loaded from external servers (cdn.facebook, apis.google.com).

I am running several tests so it is a pain to manually collect all asset urls. Is there a tool that I can use to load a web page and export the url for everything that was loaded?

This is firefox debugging. If I could export this to txt or csv, it would be perfect.

I tried CURL on debian CLI but I am no experct. Any tool will help, it does't have to be a plugin of Firefox / Chrome.

Best regards.

Answer 1

In Chrome you can export these data to a HAR file (it's JSON based) in one click. Go to "Network", right click and choose "Save as HAR with content".

Answer 2

Here's a free command line application to convert HAR files to CSV. Hope it helps.

http://www.yamamoto.com.ar/blog/?p=201

EDIT: added the project to GitHub:

https://github.com/spcgh0st/HarTools

Answer 3

On Windows you could use HttpWatch to do this with the free Basic Edition in IE or Firefox:

http://www.httpwatch.com/download/

The CSV export function will export the URLs and other fields to a CSV file.

** Disclaimer: This was posted by Simtec Limited the makers of HttpWatch **

Answer 4

Had the same requirement of exporting HAR files from Chrome DevTools or Firebug to do load testing with siege. Additionally, I wanted to replay POST requests too.

Choose one of these solutions:

• hardy @ https://github.com/nbibler/hardy - ruby script
• har2siege @ https://gist.github.com/photopresentr/7974747 - node.js (my script)