Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to Process Items in an Array in Parallel using Ruby (and open-uri)

I am wondering how i can go about opening multiple concurrent connections using open-uri? i THINK I need to use threading or fibers some how but i'm not sure.

Example code:

def get_doc(url)
  begin
    Nokogiri::HTML(open(url).read)
  rescue Exception => ex
    puts "Failed at #{Time.now}"
    puts "Error: #{ex}"
  end
end

array_of_urls_to_process = [......]

# How can I iterate over items in the array in parallel (instead of one at a time?)
array_of_urls_to_process.each do |url|
  x = get_doc(url)
  do_something(x)
end
like image 314
Mario Zigliotto Avatar asked Sep 26 '11 21:09

Mario Zigliotto


1 Answers

A simple method using threads:

threads = []

[1, 2, 3].each do |i|
  threads << Thread.new { puts i }
end

threads.each(&:join)
like image 61
Dorian Avatar answered Sep 30 '22 13:09

Dorian