Increase Python's stdout buffer size

Question

Is there a way to increase the stdout buffer size from 8182 in Python or to delay the flush until I actually call flush?

Things I've tried that don't work:

• I can get around this issue on Windows because I can access the buffer directly (e.g. see my answer to this post). But this doesn't work for Unix.
• I can increase the buffer size for a file by passing buffer to the constructor, however stdout is already constructed.
• Turning off buffering (python -u) obviously makes things worse!
• Using a temporary buffer encounters the same problems - stdout is flushed after every 8192nd byte is copied from the temporary buffer.

Rationale: The aim here is to reduce console flickering. Buffering everything, as per this question indeed works, for instance when I try in C or by using the Windows API in Python, but the 8182 limit in Python seems to be causing problems that I can't get around on Unix.

Answer 1

You can wrap the raw stdout stream, available as sys.stdout.buffer, with a larger buffer size with io.BufferedWriter, and then wrap the resulting buffered binary stream as a buffered text stream with io.TextIOWrapper:

import io
import sys

sys.stdout = io.TextIOWrapper(io.BufferedWriter(sys.stdout.buffer, new_size))

Here's a demonstration of the effect of increasing the buffer size of stdout to 100000 so that it does not flush the two 10000-character-long print outputs until a manual flush is called:

import io
import sys
import time

print('Original buffer size:', io.DEFAULT_BUFFER_SIZE)
for large_buffer in False, True:
    if large_buffer:
        print('Increasing buffer size...')
        sys.stdout = io.TextIOWrapper(io.BufferedWriter(sys.stdout.buffer, 100000))
    for i in range(2):
        time.sleep(2)
        print(str(i * 2) * 10000)
        time.sleep(2)
        print(str(i * 2 + 1) *10000)
        print(f'Flush #{i + 1}')
        sys.stdout.flush()

Demo: https://repl.it/@blhsing/UnkemptGullibleDecompiler