How do I pre-process an infinite stream of text before writing it to a file?

Question

I am using tcpflow to log network traffic on a server. I want to log this data to a file, but not all of it. The monitoring process would be running in the background as a daemon indefinitely.

Some lines of the stream include a byte count, and if I see that byte count (say, 800 bytes), then I need to log the next 800 bytes to the file. If not, I wish to not write to the file.

What's the best way for me to do this kind of "on-the-fly pre-processing" of the stream to decide what to redirect to the log file? Some kind of second daemon script that is listening to the stream, which gets piped in to that script?

Example:

I see the following line in the stream:

1343932842: 010.079.091.189.35856-010.104.001.199.11211: set i:1:20163484235 0 0 1429

First, I need to check that it has a "set". Then, I examine the last piece of the line (1429), then read the next 1429 bytes and write those to a file.

Answer 1

Yes, use a daemon program that takes the stream as input, and does just what you described. I would recommend C instead of a script, as it has very straightforward input/output, and very low overhead.

Assuming you have an executable called 'capture' and a filtering program called 'filter', you can chain them together from a bash shell using

bash-prompt$ capture capture-params | filter

Anything that capture writes to stdout will be available to filter as input from stdin. It is a simple matter, from filter's point of view, of reading lines, and when the end ... size pattern is found, writing the output to an output file (or again to stdout). If you write to a stdout, you can redirect that to a file using

bash-prompt$ capture capture-params | filter > output-file.txt