How to get http responses from Firefox to machine readable format in real time?

Question

I'm using a flash application in my browser to send http requests to a server. The server returns http responses to the browser. I would like to access the body/payload of these http responses with another program in real time.

I tried two approaches.

1.

I can see all the responses that Firefox receives in the Network Monitor tool that comes with Firefox. I just don't know how to access the data from any other program.

2.

I tried logging the network data in Firefox with about:networking using these log modules:

timestamp,sync,nsHttp:3

This logged the headers as expected, but omitted the bodies of the responses. Higher levels of nsHttp logged more info, but not the body.

Additional info that may be relevant:

• I cannot send requests from any other program. The flash application in the browser must make all http requests.
• I'm not stuck with Firefox, it's just my preference. A solution for a different browser is acceptable.

Answer 1

(I asume you are working on windows)

You could use fiddler to intercept the traffic between browser and server.

Fiddler acts as a Proxy and there is an add on which generates cap files from the traffic called fiddlerCap

If on a Mac or Linux you could use other command line proxys which allow you to intercept and log the traffic.

Another solution could be to make the server log its responses. This could be done be using modsecurity, check out the section about audit-log

Answer 2

The reason it was garbled when you intercepted the traffic with ngrep was because the session is encrypted.

The best way to intercept and log secure session data is to set up a MITM proxy with a fake certificate on your local machine.

For Windows, I think Fiddler is your best bet. Fiddler has a simple and effective workflow for doing this. The following documentation page, "Decrypting HTTPS-protected traffic", describes the steps involved: https://www.fiddlerbook.com/fiddler/help/httpsdecryption.asp

If you're on macOS, you could try Charles Proxy or Fiddler for Mac (which is currently still in Beta).

FireFox and Chrome also have built-in ways to log and export session data, but I think you'll find Fiddler/Charles and similar tools to be more flexible, especially if you need to consume the data elsewhere.