RuntimeError (Circular dependency detected while autoloading constant Apps multithreading

Question

I'm receiving this error: RuntimeError (Circular dependency detected while autoloading constant Apps

when I'm multithreading. Here is my code below. Why is this happening? The reason I am trying to multithread is because I am writing a HTML scraping app. The call to Nokogiri::HTML(open()) is a synchronous blocking call that takes 1 second to return, and I have 100,000+ pages to visit, so I am trying to run several threads to overcome this issue. Is there a better way of doing this?

class ToolsController < ApplicationController

def getWebsites
    t1=Thread.new{func1()}
    t2=Thread.new{func1()}
    t3=Thread.new{func1()}
    t4=Thread.new{func1()}
    t5=Thread.new{func1()}
    t6=Thread.new{func1()}
    t1.join
    t2.join
    t3.join
    t4.join
    t5.join
    t6.join
end

def func1
    puts Thread.current
    apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
    while apps.size == 1  do
        app = apps[0]
        puts app.name
        puts app.iTunes
        doc = Nokogiri::HTML(open(app.iTunes))
        array = doc.css('div.app-links a').map { |link| 
            url = link['href'] 
            url = Domainatrix.parse(url)
            url.domain + "." + url.public_suffix
        }
        array.uniq!
        if (array.size > 0)
            app.website = array.join(', ')
            puts app.website
        else
            app.website = "NONE"
        end
        app.save
        apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
    end 
end


end

Answer 1

"require" isn't thread-safe

Change your methods so that everything that is to be "required" is done so before the threads start.

For example:

def get_websites
    # values = Apps.all # try uncommenting this line if a second-try is required

    ar = Apps.where("apps.website IS NULL")

    t1 = Thread.new{ func1(ar) }
    t2 = Thread.new{ func1(ar) }

    t1.join
    t2.join
end

def func1( ar )
    apps = ar.order("RANDOM()").limit(1)

    while (apps.size == 1)
      puts Thread.current
    end 
end

But as somebody pointed out, the way you're multithreading within the controller isn't advised.