Threading in Ruby. Alternative to ThreadPool

Question

Lets say I have 2 different tables that needed to be filled. Each table can have different number of source files. I want to fill the tables in parallel but each source file will be read consequently. example: Table A: 3 source files, Table B: 1 source files

tables = {'A': [sourcefileA_1, sourcefileA_2,sourcefileA_3], 'B':[sourcefileB_1]}
threads = []
tables.each do |table,source_files|
 threads << Thread.new { do_something(table, source_files)}
end
threads.each(&:join)
def do_something(table, source_files)
 source_files.each do |s_file|
 #do something
end

So, the number of threads depends on the number of tables I have. Lets say now I have 500.000 tables and just a normal Mac Pro. I thought of using ThreadPool, however, if some error occurs in the thread it is not shown. In comparison if I use just threads the error is still shown.Is there any gem that can help me to use thread pool and that will throw an error if it occurs in one of the threads.

Regarding not showing the error, consider following:

require 'thread/pool' # from the ruby-thread gem
pool = Thread.pool(4)
pool.process do 
 puts(b)
end

b is not specified but it does not show me the normal error message.

Answer 1

First,

check this out (the answer) about error handling.

second, there is a nice gem called concurrent-ruby which have many aspects of multi threading in other languages implemented in ruby.

Another issue to consider is that ruby is not really concurrent, because of the GIL. But you got fibers, which do good job for your case, async IO.