How to make nginx print full log for tcp stream

Question

I'm using nginx-1.11.8 with the following configuration.

stream {

    log_format  basic   '$time_iso8601 $remote_addr '
                        '$protocol $status $bytes_sent $bytes_received '
                        '$session_time $upstream_addr '
                        '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log      logs/stream.log  basic buffer=1k flush=5s;

    include *.stream.conf;
}

Now I can only get IP and other unimportant stuff in the tcp log. There are some important information related to the IP in the tcp data packet that I want to know. What should I do to get the full tcp packet in the tcp log? Thanks in advance.

Answer 1

I don't think this is possible with stock nginx, although it may someday be possible with openresty. The ngx_stream_lua_module is still in its infancy and may yet develop features to support peeking at stream content. They might be accepting feature requests, too.

As to why e.g. $request_body doesn't exist in stream context today, I think there are a few reasons:

• Stream support is still somewhat new, compared to HTTP support
• Nginx doesn't speak the underlying protocol (of course, you could put HTTP traffic through a stream block but nginx won't handle it any differently than non-HTTP traffic)
• As a consequence, there are questions nginx can't categorically answer, like:
  1. What is a request?
  2. Where does it start?
  3. Where does it end?
  4. Does it even have something like a "body"?
  5. Is there even such a thing as a "request"?
• In http context, nginx already needed to implement buffering of requests to support retrying them with multiple server lines in an upstream block when one fails. It's not safe to do this with arbitrary TCP protocols (heck, it's not even safe to do this with HTTP in all circumstances, but that's the server admin's concern). No such feature exists presently in stream context, so there's no buffer to piggy back off of for a variable.

For some elaboration on those questions nginx can't answer, which may seem silly at first, consider that TCP is a pretty generic protocol. The simplest application layer protocols built on top of TCP define a very basic request-response model where one end sends a plain text request followed by a delimiter, while the other side waits for the request then sends a plain text response followed by a delimiter, then one or both sides close the connection. In those cases, sure, it seems straightforward to capture and log the request and response.

However, lots of more complicated protocols don't work that way. Some protocols don't even operate on a request-reply model at all and instead are asynchronous, where either side can send a "message" at any time, like WebSockets, STOMP, or AMQP. There may be no delimiters in the stream at all, or the delimiters could be hidden under layers of compression and encryption, and not all network encryption is SSL/TLS. Moreover, what if the content is encoded as ASN.1, Protobuf, Thrift, Avro, or something like that? You'd probably want a nice pretty printed form but nginx can't do that for you, and dumping a bunch of binary data in the log will be difficult if not impossible to reliably parse. There could also be nested "channels" or "sessions" like with SSH. Heck, even with request-response models, which side makes the requests and which side sends the responses could get flipped over the lifetime of the connection.

All that having been said, I think there is an argument for handling the common or at least simple cases, so maybe this will make it into nginx some day. Until then, it's probably easiest to put an intermediary either in front of or behind nginx using socat or another network debugging tool.