Real Time updating executable subprocess console output in Python

Question

I have been reading this question, and this one and noted that the solutions proposed there did not do the trick I'm trying to pull off. The idea is similar to this question but I'm giving a better example here.

I need to build a GUI in Python 3.7 to monitor and control the execution of a legacy software written in C. Right now, I'm trying to simply call a test executable from Python and have it printed on a GUI, but for now it would suffice to print on the same console as python. The problem is that the executable takes very long to fully execute, but prints console messages in the meantime, I need these messages to be read promptly by my GUI.

Here is a working example:

in Python 3.7:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        subprocess.Popen(["hello.exe"], stdout=sys.stdout, bufsize=1)

class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

Now, the example executable ("hello.exe") can be build in C like this:

#include <stdio.h>
#include <time.h>

int main() {
    struct timespec time;
    double tic = 0;
    double toc = 0;

    for( int ii = 0; ii < 3 ; ii++){
        clock_gettime(CLOCK_MONOTONIC, &time);
        tic = time.tv_sec;
        toc = tic;

        while(toc-tic<3) {
        clock_gettime(CLOCK_MONOTONIC, &time);
        toc    = time.tv_sec;
        }
        printf("Hello #%d \n",ii);
    }
    return(0); 
}

Ideally, I need the messages "Hello" and "Hi back" to be interleaved. But if I run the python script above, I get:

Hi back (1)!
Hi back (2)!
Hi back (3)!
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #0 
Hello #1 
Hello #2 
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

Apparently, the output from the executable is only printed when the execution is finished. This means that if the legacy executable was running, only upon being finished anything would show up.

Edit: I do not have hard real time constraints on this, if printing actually takes a few minutes, that's fine, the problem is that if a process needs to run for days, then every few hours (preferably minutes) the GUI needs to be updated with the console prints from the executable.

Answer 1

Consider reading Popen.stdout:

https://docs.python.org/3/library/subprocess.html?highlight=subprocess#subprocess.Popen.stdout

Here is the output of the updated program:

$ python test.py
Hi back (1)!
Hi back (2)!
Hi back (3)!
Hello #0
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hello #1
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #2
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

The updated program:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        proc = subprocess.Popen(["hello.exe"], stdout=subprocess.PIPE)
        for line in proc.stdout:
            print(line.strip().decode('utf-8'))


class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

EDIT:

I also modified C program to flush stdout after printing, buffered stdout seems to be the root cause of the problem:

    printf("Hello #%d \n",ii);
    fflush(stdout);