I have a Rails app in which I have a Rake task that uses multithreading functions supplied by the concurrent-ruby gem.
From time to time I encounter Circular dependency detected while autoloading constant
errors.
After Googling for a bit I found this to be related to using threading in combination with loading Rails constants.
I stumbled upon the following GitHub issues: https://github.com/ruby-concurrency/concurrent-ruby/issues/585 and https://github.com/rails/rails/issues/26847
As explained here you need to wrap any code that is called from a new thread in a Rails.application.reloader.wrap do
or Rails.application.executor.wrap do
block, which is what I did. However, this leads to deadlock.
The recommendation is then to use ActiveSupport::Dependencies.interlock.permit_concurrent_loads
to wrap another blocking call on the main thread. However, I am unsure which code I should wrap with this.
Here's what I tried, however this still leads to a deadlock:
@beanstalk = Beaneater.new("#{ENV.fetch("HOST", "host")}:#{ENV.fetch("BEANSTALK_PORT", "11300")}")
tube_name = ENV.fetch("BEANSTALK_QUEUE_NAME", "queue")
pool = Concurrent::FixedThreadPool.new(Concurrent.processor_count * 2)
# Process jobs from tube, the body of this block gets executed on each message received
@beanstalk.jobs.register(tube_name) do |job|
ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
@logger.info "Received job: #{job.id}"
Concurrent::Future.execute(executor: pool) do
Rails.application.reloader.wrap do
# Stuff that references Rails constants etc
process_beanstalk_message(job.body)
end
end
end
end
@beanstalk.jobs.process!(reserve_timeout: 10)
Can anyone shed a light as to how I should solve this? The odd thing is I encounter this in production while other information on this topic seems to imply it should normally only occur in development.
In production I use the following settings:
config.eager_load = true
config.cache_classes = true
.
Autoload paths for all environments are Rails default plus two specific folders ("models/validators" & "jobs/concerns").
eager_load_paths
is not modified or set in any of my configs so must be equal to the Rails default.
I am using Rails 5 so enable_dependency_loading
should equal to false
in production.
You likely need to change your eager_load_paths
to include the path to the classes or modules that are raising the errors. eager_load_paths
is documented in the Rails Guides.
The problem you're running into is that Rails is not loading these constants when the app starts; it automatically loads them when they are called by some other piece of code. In a multithreaded Rails app, two threads may have a race condition when they try to load these constants.
Telling Rails to eagerly load these constants means they will be loaded once when the Rails app is started. It's not enough to say eager_load = true
; you have to specify the paths to the class or module definitions as well. In the Rails application configuration, this is an Array
under eager_load_paths
. For example, to eager load ActiveJob
classes:
config.eager_load_paths += ["#{config.root}/app/jobs"]
Or to load a custom module from lib/
:
config.eager_load_paths += ["#{config.root}/lib/custom_module"]
Changing your eager load settings will affect the behavior of Rails. For example, in the Rails development
environment, you're probably used to running rails server
once, and every time you reload one of the endpoints it will reflect any changes to code you've made. That will not work with config.eager_load = true
, because the classes are loaded once, at startup. Therefore, you will typically only change your eager_load
settings for production
.
Update
You can check your existing eager_load_paths
from the rails console
. For example, these are the default values for a new Rails 5 app. As you can see, it does not load app/**/*.rb
; it loads the specific paths that Rails is expected to know about.
Rails.application.config.eager_load_paths
=> ["/app/assets",
"/app/channels",
"/app/controllers",
"/app/controllers/concerns",
"/app/helpers",
"/app/jobs",
"/app/mailers",
"/app/models",
"/app/models/concerns"]
In my gems (i.e., in plezi
and iodine
) I solve this with if
statements, mostly.
You'll find code such as:
require 'uri' unless defined?(::URI)
or
begin
require 'rack/handler' unless defined?(Rack::Handler)
Rack::Handler::WEBrick = ::Iodine::Rack # Rack::Handler.get(:iodine)
rescue Exception
end
I used these snippets because of Circular dependency detected
warnings and errors.
I don't know if this helps, but I thought you might want to try it.
I had this issue while trying out two gems that handles parallel processing;
For pmap I kept getting an error related to Celluloid::TaskTerminated and for parallel I was getting a Circular dependency detected while autoloading constant for when I ran it with more than 1 thread. I knew this issue was related to how my classes and modules were eager loading and race to be placed on a thread. I try enabling both of the configs to true config.cache_classes = true
and config.eager_load = true
in the development env and that did the trick for me.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With