Streaming data from STDOUT

Question

So I have the following code:

reader.rb

require 'open4'

def streamer(stdout)
  begin
    loop do
      data = stdout.read_nonblock(8)
      print data
    end
  rescue Errno::EAGAIN
    retry
  rescue EOFError
    puts 'End of file'
  end
end
pid, stdin, stdout, stderr = Open4::popen4 "ruby threader.rb"
stdout.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK)
streamer(stdout)

threader.rb

10.times do
  $stdout.puts "test"
  sleep 1
end

One ruby script is simple a spinner that puts to stdout every second.

The other is meant to run that script and I would like to capture the data as it comes in. So I want the stream to read from stdout non-blocking.

I cannot seem to get this to work. I think I am setting the fctnl O_NONBLOCK flag properly but perhaps I am not.

Answer 1

Your code is working fine. You're only missing one piece and that is flushing the output buffer of your threader.

The problem is STDOUT is almost always buffered, and in this case the buffer will not be flushed, unless explicitly told to, until after threader exits.

That is why the reader will not see anything, and then suddenly it will get a burst of output when threader exits and STDOUT gets flushed.

So the simple fix for this test is:

10.times do
  $stdout.puts "test"
  $stdout.flush
  sleep 1
end

Note however that because you are reading using NONBLOCK your reader will busy loop(!) consuming 100% CPU. What you really should do to prevent busylooping is wait for incoming data before reading.

This can be done with IO.select:

...
loop do
  IO.select([stdout]) # <- waits for data (any data, even 1 byte)
  data = stdout.read_nonblock(8)
  print data
end  
...