How to capture stdout from another process in Win32 without latency?

Question

What I'd like to do is similar to what Visual Studio does in its output window or other editors in their tool windows: Start another process B from my process A and capture its stdout/stderr output.

So far, I got it working with CreatePipe(), but for some reason, the output of B doesn't arrive at B right when it gets written. It behaves more like a buffer of some kind gets filled and when it's full, all the buffers content arrives at A at once. I wrote my own test program that outputs something and does an fflush(stdout) directly afterwards. Then the output directly arrives at A. But I can't change the code of all the B processes I'd like to use that way. Trying to flush the pipe from A also doesn't work.

How is this supposed to work?

My initialization code as well as consuming code:

 sa.nLength = sizeof(SECURITY_ATTRIBUTES);
 sa.bInheritHandle = TRUE;
 sa.lpSecurityDescriptor = NULL;

 err = CreatePipe(&hChildStdoutRd, &hChildStdoutWr, &sa, stdouthistory);
 if (err == 0)
     return 1;
 err = DuplicateHandle(GetCurrentProcess(), hChildStdoutRd,
                       GetCurrentProcess(), &hChildStdoutRdDup , 0,
                       FALSE,
                       DUPLICATE_SAME_ACCESS);
 if (err == 0)
     return 3;
 CloseHandle(hChildStdoutRd);

 DWORD a, b, c;
 a = PIPE_READMODE_BYTE | PIPE_NOWAIT;
 b = 0;
 c = 0;
 SetNamedPipeHandleState(hChildStdoutRdDup, &a, &b, &c);

 err = CreatePipe(&hChildStdinRd, &hChildStdinWr, &sa, stdinhistory);
 if (err == 0)
     return 1;
 err = DuplicateHandle(GetCurrentProcess(), hChildStdinWr,
                       GetCurrentProcess(), &hChildStdinWrDup , 0,
                       FALSE,
                       DUPLICATE_SAME_ACCESS);
 if (err == 0)
     return 4;
 CloseHandle(hChildStdinWr);

 a = PIPE_READMODE_BYTE | PIPE_NOWAIT;
 b = 0;
 c = 0;

 ZeroMemory(&si,sizeof(STARTUPINFO));
 si.cb = sizeof(STARTUPINFO);
 si.dwFlags = STARTF_USESTDHANDLES;
 si.wShowWindow = SW_SHOW;

 si.hStdOutput = hChildStdoutWr;
 si.hStdError = hChildStdoutWr;
 si.hStdInput = hChildStdinRd;

 ZeroMemory( &pi, sizeof(PROCESS_INFORMATION) );

 err = CreateProcess(0, this->cmdline, 0, 0, true, CREATE_NO_WINDOW, 0, 0, &si, &pi);
 if (err == 0)
     return 4;

Consumption:

 DWORD avail;
 unsigned int ofs = 0;
 if (PeekNamedPipe(hChildStdoutRdDup, NULL, 0, NULL, &avail, NULL))
 {
     if (avail != 0)
     {
         int err = ReadFile(hChildStdoutRdDup, s + ofs, slen, &threadbuffern, 0);
                           // Consume ...
     }
 }

Edit: I just found this question: Continuously read from STDOUT of external process in Ruby. It's the same problem, but in the context of Ruby. Sadly the solution was to use some Ruby library that just makes it work. How does that library do it? What is the equivalent in Win32/C++?

Answer 1

You can't do that. If the output has not been flushed in the offending process, it's not actually been written to stdout in the first place. That is, the OS has not actually even gotten the data from the target process yet.

This isn't any kind of inherent latency with pipes, it's that the programs you're monitoring haven't actually written it into the pipe yet.

You should notice the exact same behavior for the command prompt when executing said programs, because the command prompt uses the same pipe solution you're using. If you don't it's because the programs in question are detecting that they're writing to a file handle, rather than a console handle, and doing additional buffering.