inotify + stdout piping - output being lost in a pipe

Question

I have some one-liner generating events for inotify.

while true; do for i in $(seq 1 100); do touch /tmp/ino/foo$i; sleep 1s; done; rm /tmp/ino/foo*; done

I then set up a small bash pipeline to watch that folder, ignoring events about ISDIR (maybe I could do that with inotifywait, but that's not relevant):

inotifywait -m -e close /tmp/ino 2>/dev/null | grep -v ISDIR

And that works fine, I see lines like /tmp/ino/ CLOSE_WRITE,CLOSE foo57.

But if I add an extra pipe at the end, I don't get any output. To keep it simple, let's use the fact that grep pattern is idempotent.

inotifywait -m -e close /tmp/ino 2>/dev/null | grep -v ISDIR | grep -v ISDIR

This produces no output. I know the my generator is still running, and a pipeless inotifywait -m -e close /tmp/ino in another terminal is still producing output.

After a bit of thinking, I assumed it was probably a buffering problem (issues like this often seem to be). I changed my pipeline to

inotifywait -m -e close /tmp/ino 2>/dev/null | grep -v ISDIR --line-buffered | grep -v ISDIR

And now I'm getting output again, so that fixes the problem.

However, I don't really understand why it failed to work without forcing line buffering. I've never experienced issues like this with grep, even with 'slow producing' outputs. However, some other programs in the pipeline, forcing sed to be sed -u , and forcing me to add fflush() in at the end of each awk statement.

So, what's forcing strange buffering here, and how can I fix it (without having to scrabble around in man pages looking for esoteric force line buffering commands)?

Answer 1

inotifywait is probably buffering. I would have suggested using stdbuf:

stdbuf -oL inotifywait -m -e close /tmp/ino 2>/dev/null | grep ...