Ruby: Read large data from stdout and stderr of an external process on Windows

Question

Greetings, all,

I need to run a potentially long-running process from Ruby 1.9.2 on Windows and subsequently capture and parse the data from the external process's standard output and error. A large amount of data can be sent to each, but I am only necessarily interested in one line at a time (not capturing and storing the whole of the output).

After a bit of research, I found that the Open3 class would take care of executing the process and giving me IO objects connected to the process's standard output and error (via popen3).

Open3.popen3("external-program.bat") do |stdin, out, err, thread|
  # Step3.profit() ?
end

However, I'm not sure how to continually read from both streams without blocking the program. Since calling IO#readlines on out or err when a lot of data has been sent results in a memory allocation error, I'm trying to continuously check both streams for available input, but not having much luck with any of my implementations.

Thanks in advance for any advice!

Answer 1

After a lot of different trial and error attempts, I eventually came up with using two threads, one to read from each stream (generator.rb is just a script I wrote to output things to standard out and err):

require 'open3'

data = {}

Open3.popen3("ruby generator.rb") do |stdin, out, err, external|
  # Create a thread to read from each stream
  { :out => out, :err => err }.each do |key, stream|
    Thread.new do
      until (line = stream.gets).nil? do
        data[key] = line
      end
    end
  end

  # Don't exit until the external process is done
  external.join
end

puts data[:out]
puts data[:err]

It simply outputs the last line sent to standard output and error by the calling program, but could obviously be extended to do additional processing (with different logic in each thread). A method I was using before I finally came up with this was resulting in some failures due to race conditions; I don't know if this code is still vulnerable, but I've yet to experience a similar failure.