How to read stdout unbuffered with Open#popen3 in Ruby

Question

I'm running a process using popen3, and trying to read the stdout stream. I want to be able to detect each character as it's put into stdout, but even using stdout.getc, I'm only able to get any output once there's a newline.

Any tips? I've seen this answered with stdin.getc, but not stdout.

Answer 1

The problem you have is that most programs when run through a pipe, which is what happens with popen, will be run using buffered output.

But what you are looking for is unbuffered output. Unbuffered output only happens when the proccess is attached to a PTY. I.e. a shell.

When you attach the process to a pipe the output will be buffered unless the process is explicitly calling flush during its output processing.

You have a few options:

1. If you have control over the source code that you run with popen then force a flush call on stdout after each output sequence.

2. You could try and run the command under stdbuf -o0, which tries to force unbuffered output.
   See this answer:
   Force line-buffering of stdout when piping to tee.
   However that is not guaranteed to work for all programs.

3. You could try to use the Ruby PTY library instead of popen to make the program run under a pseudo-terminal, and hence run unbuffered. See this answer:
   Continuously read from STDOUT of external process in Ruby

Option 3 is most likely to work with any program you want to run and monitor, while the other options may or may not work depending on the program you are running and if you have access to the source code of that program.