Limited buffer in Popen [duplicate]

Question

I am launching a script using a python code. This code should launch the script which writes a file on the disk, and wait for the script to finish.

But whenever I launch this script using python, the result file doesn't exceed 65768 bytes and the script doesn't respond anymore. Here is what I use in python :

p = subprocess.Popen(command, 
                     shell=True, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=-1)
p.wait()

where command is the command for the script.

Does anyone know a solution for this issue ?

Thank you.

Answer 1

from time import sleep
p = subprocess.Popen(command, 
                     shell=True, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=-1)
output = ''
while p.poll() is None:
    output += p.stdout.readline()+'\n'  # <--- This is your magic
    sleep(0.025)
output += p.stdout.read() # <--- And this is just to get the leftover data
print('Command finished')

p.stdout.close()

as @J.F comments on almost every single Popen answer i give, you should never define stdout=..., stdin=... or stderr=... unless you're going to utelize them. Because they will fill up the buffer and hang your application.

But if you do, make sure you "tap" from it once in a while.