Dump the whole HTTP communication as raw data in nodejs

Question

I wonder if it's possible to dump the whole HTTP request+response as it goes over the wire.

I don't want to get the method, the path info, the query string, the headers, the cookies, the body and whatever. I could in theory assemble the raw data myself, but then I'd need no HTTP library, right?

Moreover, I want to dump exactly the bytes that go over the wire.

I want the raw data like in this image

taken from this page.

I'm using current node.js as a HTTP client with request. It's plain HTTP (no HTTPS).

Installing a proxy in node.js would be an option, but I don't insist on a library. I could imagine to wrap the socket read and write functions, but I can't see how to get to the socket used.

Answer 1

The Request module returns augmented native objects. The return value is an augmented http.ClientRequest object (kind of), and the callback is provided an augmented http.IncomingMessage as the second argument. You can use the various properties to reconstruct the response, but you can't get it directly from here. The native http API that Node provides abstracts away the raw response.

(Docs for IncomingMessage and ClientRequest are here: https://nodejs.org/api/http.html).

More interestingly, these are both abstractions over net.Socket. If you use the native http API, you can listen to this Socket before you send a ClientRequest (with .end). This will give you a Buffer containing the HTTP response.

let http = require("http");
let nativeRequest = http.get({
    host: "google.com"
}); //get a ClientRequest object
nativeRequest.on('socket', function (socket) {
    socket.on('data', function (data) { console.log(data.toString()); });
});
nativeRequest.end();

It doesn't look like this enables you to snoop on the outbound request, but it works great for the response.

Moving back up the chain of abstraction, this works fine with Request. I'll skip the snippet because it's almost identical to both the prior and the upcoming.

To get the request, we can poke around in the internals of a Socket to see if there's something we can abuse. Object.keys(socket) returns the following array:

[
   "connecting",
   "_hadError",
   "_handle",
   "_parent",
   "_host",
   "_readableState",
   "readable",
   "domain",
   "_events",
   "_eventsCount",
   "_maxListeners",
   "_writableState",
   "writable",
   "allowHalfOpen",
   "destroyed",
   "_bytesDispatched",
   "_sockname",
   "_pendingData",
   "_pendingEncoding",
   "server",
   "_server",
   "parser",
   "_httpMessage"
]

And, indeed, if we poke at the suspicious looking _pendingData, we can view the request before it is sent:

let request = require('request');

let req = request("http://google.com", function (e, r, d) {});
req.on('socket', function (socket) {
    console.log("========\nRequest\n========")
    console.log(JSON.stringify(socket._pendingData, null, 3));
    console.log("========\nResponse\n========");
    socket.on('data', function (data) { console.log(data.toString()); });
});