Why I do not see realtime output in the output file?

Question

I thought writing a file gives real-time output, since it is so when I use C/C++ to write files. But when I run python program it seems the output file is always 0 byte until the whole program finished running. Even for the nohup python xxx.py &, the print stuff in the file nohup.out isn't realtime, and can only be seen after execution. I'm now running really big program and want to see the progress in the file, how can I achieve it?

Answer 1

There are two points at which your file can buffer - Python's internal buffers and the buffers on the operating system. This is a performance boost that avoids system calls and disk writes while the buffer is filling up.

Calling file.flush() will push the internal buffer to the operating system. You can additionally call fsync to request the operating system to save to disk.

Usually you can leave the operating system to do what it knows best, so calling flush is usually enough for most applications. The same is partially true for Python's internal buffer - it knows best in terms of performance, but you may require more frequent writes and be willing to pay the additional cost. The only way to know the exact cost is to measure it both ways.