How does/frequent unix tee command write stdout terminal output to file? if the output is too big

Question

I am redirecting some tool stdout to tee command so that current progress can be seen on terminal as well as in the log file

Here is the code snippet where I am running tool and its stdout is fed to tee command and this code snippet is written from tcl script.

$(EH_SUBMIT) $(ICC_EXEC) $(OPTIONS) -f ./scripts/[email protected] | tee -i ./logs/[email protected]

I can see current real time progress on the terminal but the same observation is not seen in the log file! and it writes stdout to log file chunk by chunk

How does tee work? Does it write by blocks or time or both? If block what is the minimum block size? If it is time what is minimum duration?

I need to parse real time log entries for some data analytics(as I read log file via tail -f and then push new data as the log file grows).

Answer 1

Unless programs handle buffering on their own, buffering of IO streams is handled in the libc. The standard behaviour is: Buffer output line wise if it goes to a terminal, buffer output block wise if it goes to a non-terminal, meaning a file or a pipe. That's why the output appears in log file as you described it: chunk by chunk. This behaviour is for performance optimization.

On Linux the stdbuf command can be used to run a program with adjusted buffers. You need to run your program like this:

stdbuf -oL your_program >> your.log &
tail -f your.log

-oL means buffer stdout linewise.