How can I read process output that has not been flushed?

Question

Consider this little programm be compiled as application.exe

#include <stdio.h>

int main()
{
    char str[100];
    printf ("Hello, please type something\n");
    scanf("%[^\n]s", &str);
    printf("you typed: %s\n", str);
    return 0;
}

Now I use this code to start application.exe and fetch its output.

#include <stdio.h>
#include <iostream>
#include <stdexcept>

int main()
{
    char buffer[128];
    FILE* pipe = popen("application.exe", "r");
    while (!feof(pipe)) {
        if (fgets(buffer, 128, pipe) != NULL)
            printf(buffer);
    }
    pclose(pipe);
    return 0;
}

My problem is that there is no output until I did my input. Then both output lines get fetched. I can workarround this problem by adding this line after the first printf statement.

fflush(stdout);

Then the first line is fetched before I do my input as expected.

But how can I fetch output of applications that I cannot modify and that do not use fflush() in "realtime" (means before they exit)? . And how does the windows cmd do it?

Answer 1

You have been bitten by the fact that the buffering for the streams which are automatically opened in a C program changes with the type of device attached.

That's a bit odd — one of the things which make *nixes nice to play with (and which are reflected in the C standard library) is that processes don't care much about where they get data from and where they write it. You just pipe and redirect around at your leisure and it's usually plug and play, and pretty fast.

One obvious place where this rule breaks is interaction; you present a nice example. If the output of the program is block buffered you don't see it before maybe 4k data has accumulated, or the process exits.

A program can detect though whether it writes to a terminal via isatty() (and perhaps through other means as well). A terminal conceptually includes a user, suggesting an interactive program. The library code opening stdin and stdout checks that and changes their buffering policy to line buffered: When a newline is encountered, the stream is flushed. That is perfect for interactive, line oriented applications. (It is less than perfect for line editing, as bash does, which disables buffering completely.)

The open group man page for stdin is fairly vague with respect to buffering in order to give implementations enough leeway to be efficient, but it does say:

the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device.

That's what happens to your program: The standard library sees that it is running "non-interactively" (writing to a pipe), tries to be smart and efficient and switches on block buffering. Writing a newline does not flush the output any longer. Normally that is a good thing: Imagine writing binary data, writing to disk every 256 bytes, on average! Terrible.

It is noteworthy to realize that there is probably a whole cascade of buffers between you and, say, a disk; after the C standard library comes the operating system's buffers, and then the disk's proper.

Now to your problem: The standard library buffer used to store characters-to-be-written is in the memory space of the program. Despite appearances, the data has not yet left your program and hence is not (officially) accessible by other programs. I think you are out of luck. You are not alone: Most interactive console programs will perform badly when one tries to operate them through pipes.

Answer 2

IMHO, that is one of the less logical parts of IO buffering: it acts differently when directed to a terminal or to a file or pipe. If IO is directed to a file or a pipe, it is normally buffered, that means that output is actually written only when a buffer is full or when an explicit flush occurs => that is what you see when you execute a program through popen.

But when IO is directed to a terminal, a special case occurs: all pending output is automatically flushed before a read from the same terminal. That special case is necessary to allow interactive programs to display prompts before reading.

The bad thing is that if you try to drive an interactive application through pipes, you loose: the prompts can only be read when either the application ends or when enough text was output to fill a buffer. That's the reason why Unix developpers invented the so called pseudo ttys (pty). They are implemented as terminal drivers so that the application uses the interactive buffering, but the IO is in fact manipulated by another program owning the master part of the pty.

Unfortunately, as you write application.exe, I assume that you use Windows, and I do not know an equivalent mechanism in the Windows API. The callee must use unbuffered IO (stderr is by default unbuffered) to allow the prompts to be read by a caller before it sends the answer.