Menu
I have a program that writes to fd3 and I want to process that data with grep and sed. Here is how the code looks so far:
exec 3> >(grep "good:"|sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
Nothing is output until I do a
exec 3>&-
Then, everything that I wanted finally arrives as I expected:
I got: data2
It seems to reply immediately if I use only a grep or only a sed, but mixing them seems to cause some sort of buffering. How can I get immediate output from fd3?
783
env | sed '/^#/ d' | sed '/^$/ d' Concatenate FILE(s), or standard input, to standard output. With no FILE, or when FILE is -, read standard input.
The sed command ran in about 12 seconds which I never would have believed (working with a normal HDD). Within 12 seconds the command read through 30 Gb of text, truncating each file to only keep the respective lines I was filtering for.
unix is a powerful. Replacing all the occurrence of the pattern in a line : The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.
In sed, p prints the addressed line(s), while P prints only the first part (up to a newline character \n ) of the addressed line. If you have only one line in the buffer, p and P are the same thing, but logically p should be used.
You only need to tell grep and sed to not bufferize lines:
grep --line-buffered
and
sed -u
Turn off buffering in pipe seems to be the easiest and most generic answer. Using stdbuf (coreutils) :
exec 3> >(stdbuf -oL grep "good:" | sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
I got: data2
Buffering has other dependencies, for example depending on mawk either gawk reading this pipe :
exec 3> >(stdbuf -oL grep "good:" | awk '{ sub(".*:", "I got: "); print }')
In that case, mawk would retain the input, gawk wouldn't.
See also How to fix stdio buffering
42
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
