grep with continuous pipe does not work

Question

(maybe it is the "tcpflow" problem)

I write a script to monitoring http traffic, and I install tcpflow, then grep

it works (and you should make a http request, for example curl www.163.com)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '

it outputs like this (continuously)

Host: config.getsync.com
Host: i.stack.imgur.com
Host: www.gravatar.com
Host: www.gravatar.com

but I can't continue to use pipe

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7- 

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H

When I replace sudo tcpflow with cat foo.txt, it works:

cat foo.txt | grep '^Host: ' | grep H

so what's wrong with pipe or grep or tcpflow ?

---

update:

This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh

Answer 1

To grep a continuous stream use --line-buffered option:

sudo tcpflow -p -c -i eth0 port 80 2> /dev/null | grep --line-buffered '^Host'

--line-buffered

Use line buffering on output. This can cause a performance penalty.

---

Some reflections about buffered outputting(stdbuf tool is also mentioned):

Pipes, how do data flow in a pipeline?

Answer 2

I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '

With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 4096¹ byte chunks before sending to next command in pipeline, which is what overriden using stdbuf

---

_{1. Refer this nice detail into the subject.}