How do I HTTP post stream data from memory in Ruby?

Question

I would like to upload data I generated at runtime in Ruby, something like feeding the upload from a block.

All examples I found only show how to stream a file that must be on disk prior to the request but I do not want to buffer the file.

What is the best solution besides rolling my own socket connection?

This a pseudocode example:

post_stream('127.0.0.1', '/stream/') do |body|
  generate_xml do |segment|
    body << segment
  end
end

Answer 1

Code that works.

    require 'thread'
    require 'net/http'
    require 'base64'
    require 'openssl'

    class Producer
      def initialize
       @mutex = Mutex.new
       @body = ''
       @eof = false
      end

      def eof!()
        @eof = true
      end

      def eof?()
        @eof
      end

      def read(size)
        @mutex.synchronize {
          @body.slice!(0,size)
        }
      end

      def produce(str)
        if @body.empty? && @eof
          nil
        else
          @mutex.synchronize { @body.slice!(0,size) }
        end
      end
    end

    data = "--60079\r\nContent-Disposition: form-data; name=\"file\"; filename=\"test.file\"\r\nContent-Type: application/x-ruby\r\n\r\nthis is just a test\r\n--60079--\r\n"

    req = Net::HTTP::Post.new('/')
    producer = Producer.new
    req.body_stream = producer
    req.content_length = data.length
    req.content_type = "multipart/form-data; boundary=60079"

    t1 = Thread.new do
      producer.produce(data)
      producer.eof!
    end

    res = Net::HTTP.new('127.0.0.1', 9000).start {|http| http.request(req) }
    puts res

Answer 2

There's a Net::HTTPGenericRequest#body_stream=( obj.should respond_to?(:read) )

You use it more or less like this:

class Producer
  def initialize
   @mutex = Mutex.new
   @body = ''
  end

  def read(size)
    @mutex.synchronize {
      @body.slice!(0,size)
    }
  end

  def produce(str)
    @mutex.synchronize {
      @body << str
    }
  end
end

# Create a producer thread

req = Net::HTTP::Post.new(url.path)
req.body_stream = producer
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }