Rails 4.2 Autoloading not thread-safe

Question

I have the following model:

class User < ActiveRecord::Base
  def send_message(content)
    MessagePoro.new(content).deliver!
  end

  def self.send_to_all(content)
    threads = []
    all.each do |user|
      threads << Thread.new do
        user.send_message(content)
      end
    end
    threads.each(&:join)
  end
end

MessagePoro model can be something simple, e.g. app/models/message_poro.rb:

class MessagePoro
  def initialize(content)
    # ...
  end

  def deliver!
    # ...
  end
end

Now, when I'm having e.g. 100 Users, and I'm running User.send_to_all("test") I'm sometimes getting thoser errors:

RuntimeError: Circular dependency detected while autoloading constant MessagePoro

or:

wrong number of arguments (1 for 0)

I figured it must be because MessagePoro isn't loaded and all Threads try to load it simultaneously, or something like that. Since those errors occur only sometimes, I'm pretty sure it's only when there is a 'race condition' or has something todo with Threading. I have tried to initialize MessagePoro before starting the Threads, and I have played around with eager_loading, but the problem seems to persist. What else can I try to mitigate this issue?

Answer 1

I recently ran into a very similar issue when trying to use an extra custom library placed in the [rails_root]/lib directory.

TL;DR:

You can use eager loading to around the issue, as that makes sure all constants/modules/classes are in memory before any actual code runs. However for this to work:

1. You must have config.eager_load = true set in the Rails config (this is done by default in the Production environment)
2. The file that your class-to-be-eager-loaded is in must be in the config.eager_load_paths, as opposed to config.autoload_paths.

OR

You can use require or require_dependency (another ActiveSupport feature) to make sure the code you need it explicitly loaded before it would otherwise get autoloaded by Rails.

More info

As digidigo mentioned in his reply, the circular dependency error comes from the ActiveSupport::Dependencies module, or the Rails autoloader in more general terms. This code is not threadsafe, as it uses that class/module variable to store files that it is loading. If two threads end up autoloading the same thing at the same time, one of them can get mislead by seeing the file to load already in that class variable and throwing a 'circular dependency' error.

I ran into this issue when running Rails in production mode with the (threaded) Puma webserver. We had added a small library to the lib directory in our Rails root, and initially added lib to config.autoload_once_paths. Everything was fine in Development, but in Production (with config.eager_load and config.cache_classes enabled), very occasionally we would get these same circular dependency issues with near-simultaneous requests. A few hours of debugging later, I ended up seeing the non-thread-safety happening in front of my eyes, when stepping through the ActiveSupport code around the circular dependency and seeing the different threads pick up at different points in the code. The first thread would add the file to load into the loading array, then the second thread would find it there and raise the circular dependency error.

It turns out adding something to autoload_paths or autoload_once_paths does NOT also mean that it will get picked up by eager loading. However the opposite is true - paths added to eager_load_paths will be considered for autoloading if eager_load is disabled (see this article for more info). We switched to eager_load_paths and have had no further issues so far.

Interestingly enough, just before the Rails 4 beta, autoloading was disabled in the Production environment by default, which meant that an issue like this would have caused a hard fail 100% of the time, rather than a quirky threading fail 5% of the time. However this was reverted in time for the 4.0 beta release - you can see some (passionate) discussion about it here (including the choice phrase 'honestly, you're telling me to go f*** myself?'). Since then though, that revert has been reverted ahead of the Rails 5.0.0beta1, so hopefully less people will have to deal with this headache of an issue again in the future.

Extra notes:

The Rails autoloader is totally separate from the Ruby autoloader - this seems to be because Rails does more inference on directory structure when trying to autoload constants.

Ruby's autoload appears to have been made threadsafe as of Ruby 2.0, however this has nothing to do with the Rails autoloading code. Rails's autoloader appears to be definitely not threadsafe, as previously mentioned.