Why is IO::WaitReadable being raised differently for STDOUT than STDERR?

Question

Given that I wish to test non-blocking reads from a long command, I created the following script, saved it as long, made it executable with chmod 755, and placed it in my path (saved as ~/bin/long where ~/bin is in my path).

I am on a *nix variant with ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.0.0] compiled with RVM defaults. I do not use Windows, and am therefore unsure if the test script will work for you if you do.

#!/usr/bin/env ruby

3.times do
  STDOUT.puts 'message on stdout'
  STDERR.puts 'message on stderr'
  sleep 1
end

Why does long_err produce each STDERR message as it is printed by "long"

def long_err( bash_cmd = 'long', maxlen = 4096)
  stdin, stdout, stderr = Open3.popen3(bash_cmd)
  begin
    begin
      puts 'err -> ' + stderr.read_nonblock(maxlen)
    end while true
  rescue IO::WaitReadable
    IO.select([stderr])
    retry
  rescue EOFError
    puts 'EOF'
  end
end

while long_out remains blocked until all STDOUT messages are printed?

def long_out( bash_cmd = 'long', maxlen = 4096)
  stdin, stdout, stderr = Open3.popen3(bash_cmd)
  begin
    begin
      puts 'out -> ' + stdout.read_nonblock(maxlen)
    end while true
  rescue IO::WaitReadable
    IO.select([stdout])
    retry
  rescue EOFError
    puts 'EOF'
  end
end

I assume you will require 'open3' before testing either function.

Why is IO::WaitReadable being raised differently for STDOUT than STDERR?

Workarounds using other ways to start subprocesses also appreciated if you have them.

Answer 1

In most OS's STDOUT is buffered while STDERR is not. What popen3 does is basically open a pipe between the exeutable you launch and Ruby.

Any output that is in buffered mode is not sent through this pipe until either:

1. The buffer is filled (thereby forcing a flush).
2. The sending application exits (EOF is reached, forcing a flush).
3. The stream is explicitly flushed.

The reason STDERR is not buffered is that it's usually considered important for error messages to appear instantly, rather than go for for efficiency through buffering.

So, knowing this, you can emulate STDERR behaviour with STDOUT like this:

#!/usr/bin/env ruby

3.times do
  STDOUT.puts 'message on stdout'
  STDOUT.flush 
  STDERR.puts 'message on stderr'
  sleep 1
end

and you will see the difference.

You might also want to check "Understanding Ruby and OS I/O buffering".