Ruby IO.popen STDOUT buffering

Question

I'm working on a script that uses IO.popen to open another program and continually read the data. It's like this:

process = IO.popen(["/the/program", "argument", "argument"])

loop do
  line = process.gets
  puts "#{line}"
end

(The actual program does more than just printing the output, obviously - that's just an example.)

The issue I'm running into is that popen seems to be buffering STDOUT from the opened process. I've confirmed this by running the program directly from a shell and through popen, side-by-side, and the Ruby one never gets one line at a time. It always gets multiple lines at a time, and is delayed.

I've tried

STDOUT.sync = true

... before popen, but that hasn't changed anything.

The program in question is definitely using \n as a new line, so that's not the issue.

Answer 1

Do you have the source to the other program? You either need to force the other program to flush its output, or make your script look like a pty (see the pty standard lib).

See this question for a good explanation of what's going on.

EDIT: pty sample code:

require 'pty'
PTY.spawn "some-command" do |r,w,p|
  loop { puts r.gets }
end

Answer 2

I suspect /the/program is buffering when it detects stdout is not a terminal - you can test by piping through cat, eg:

"/the/program" "argument" "argument" | cat

The answer above, will solve it if that is the problem, ie:

#!/usr/bin/env ruby

require 'pty'
PTY.spawn "./the-program testing one Two three" do |r,w,p|
  loop { puts "GOT: #{r.gets}" }  
end

Some languages (eg C) detect if stdout is a terminal and change to line buffered - see Is stdout line buffered, unbuffered or indeterminate by default?

As an example when it works, I used a simple bash script to output each argument and the time, one at a time, with 3 seconds in between between and the ruby script worked without a problem. I added eof detection for this example.

Modified script:

#!/usr/bin/env ruby

process = IO.popen(["./the-program", "testing", "one", "Two", "three"])

while !process.eof?
  line = process.gets
  puts "GOT: #{line}"
end

the-program contents:

#!/bin/bash

for arg
do
  echo $arg
  date
  sleep 3
done

I tried with ruby version 1.9.3 and 2.1.2

$ ruby ,p
GOT: testing
GOT: Mon Jun 16 06:19:00 EST 2014
GOT: one
GOT: Mon Jun 16 06:19:03 EST 2014
GOT: Two
GOT: Mon Jun 16 06:19:06 EST 2014
GOT: three
GOT: Mon Jun 16 06:19:09 EST 2014
$ 

If I use a C program instead, then the problem reoccurs:

#include <stdio.h>

main(int argc, char **argv)
{
        int i;

        for (i=0; i<argc; i++) {
                printf("%s\n", argv[i]);
                sleep(3);
        }
}