How to stop sed from buffering?

Question

I have a program that writes to fd3 and I want to process that data with grep and sed. Here is how the code looks so far:

exec 3> >(grep "good:"|sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3

Nothing is output until I do a

exec 3>&-

Then, everything that I wanted finally arrives as I expected:

I got: data2

It seems to reply immediately if I use only a grep or only a sed, but mixing them seems to cause some sort of buffering. How can I get immediate output from fd3?

Answer 1

You only need to tell grep and sed to not bufferize lines:

grep --line-buffered 

and

sed -u

Answer 2

Turn off buffering in pipe seems to be the easiest and most generic answer. Using stdbuf (coreutils) :

exec 3> >(stdbuf -oL grep "good:" | sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
I got: data2

Buffering has other dependencies, for example depending on mawk either gawk reading this pipe :

exec 3> >(stdbuf -oL grep "good:" | awk '{ sub(".*:", "I got: "); print }')

In that case, mawk would retain the input, gawk wouldn't.

See also How to fix stdio buffering