Inconsistent "LoadError" behavior with 'lib' namespacing/autoloading

Question

We have just created a new file in 'lib' that has spawned a series of headaches involving load errors.

/lib/response_set.rb:

module MyCompany
  class ResponseSet < Array
    ...
  end
end

/spec/lib/response_set_spec.rb

require 'spec_helper'

describe MyCompany::ResponseSet do
  describe "..." do
    ...
  end
end

Running this spec in Rspec gives us the following error when it gets to the first 'describe':

/Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:492:in `load_missing_constant': Expected /Users/my_stuff/projects/my_project/lib/response_set.rb to define ResponseSet (LoadError)
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:183:in `block in const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `each'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/backward_compatibility.rb:20:in `const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-expectations-2.5.0/lib/rspec/expectations/backward_compatibility.rb:6:in `const_missing'
    from /Users/my_stuff/projects/my_project/spec/lib/response_set_spec.rb:4:in `<top (required)>'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `block in load_spec_files'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `map'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load_spec_files'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/command_line.rb:18:in `run'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:55:in `run_in_process'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:46:in `run'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/rspec-core-2.5.1/lib/rspec/core/runner.rb:10:in `block in autorun'

HOWEVER! We have been using many other files for a long long time that have identical structure. For example, here's another one that has been working fine since it was created:

/lib/smart_set.rb

module MyCompany
  class SmartSet < Array
    ...
  end
end

And /spec/lib/smart_set_spec.rb

require 'spec_helper'

describe MyCompany::SmartSet do
  describe "..." do
    ...
  end
end

This file has the identical structure but causes no problems at all.

ResponseSet ( the problem class ) apparently has loading issues for no discernable reason. In the rails console, the first time I try to create one, I get an error, but then I can create one afterwards:

Loading development environment (Rails 3.0.4)
ruby-1.9.2-p136 :001 > rs = MyCompany::ResponseSet.new
LoadError: Expected /Users/my_stuff/projects/my_project/lib/response_set.rb to define ResponseSet
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:492:in `load_missing_constant'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:183:in `block in const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `each'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:503:in `load_missing_constant'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:183:in `block in const_missing'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `each'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/activesupport-3.0.4/lib/active_support/dependencies.rb:181:in `const_missing'
    from (irb):1
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/railties-3.0.4/lib/rails/commands/console.rb:44:in `start'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/railties-3.0.4/lib/rails/commands/console.rb:8:in `start'
    from /Users/my_stuff/.rvm/gems/ruby-1.9.2-p136@my_project/gems/railties-3.0.4/lib/rails/commands.rb:23:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'
ruby-1.9.2-p136 :002 > rs = MyCompany::ResponseSet.new
 => [] 

Also, adding

require 'response_set'

at the top of response_set_spec.rb allows those tests to run. But no such thing is necessary for smart_set_spec.rb.

The following is in place in application.rb:

config.autoload_paths += %W(#{config.root}/lib)
config.autoload_paths += Dir["#{config.root}/lib/**/"]
config.autoload_paths += Dir["#{config.root}/app/models/**/"]

Now, I understand that Rails has some sort of opinion on how file structure should match namespace structure for these types of things, and we have restructured our modules and files towards this end. It SEEMS to have fixed the issue (although we were seeing some other weird load errors for a while when we ran the full test suite - these have mysteriously gone away). Nonetheless everyone here is baffled and not a little annoyed that Rails is being so inconsistent and we would like to know why. As you can see there are two files that are identical as far as namespacing and file structure are concerned, being treated completely differently. In fact we have about a dozen other files in the top level of 'lib' with similar namespacing that have never caused any problems. Can anyone explain what the heck is going on here?

Answer 1

We had a similar issue which, after digging, turned out to be caused by the changes in Rails 3.x and autoload_paths.

Our case appeared only in test (RAILS_ENV=test). When Rails was loading the controllers, it scrambled about looking for each controller's matching model (due to a ActionController::Base.wrap_parameters set in an initializer). Eventually it wound down to the method mentioned above (load_missing_constant). Since our autoload_paths included both lib and lib/**, Rails pulled in all the files from all the subdirectories under lib. Unfortunately, it appears to ignore the implied namespace when loading from subdirectories. It expected foo/base.rb to define Base vs. Foo::Base. That seems to be the core flaw: load_missing_constant invokes search_for_file which returns any file whose name matches (e.g., in my example, foo/base.rb was returned as it matched base.rb).

It's hard to say if this is an error in Rails - in that it violates the namespace-to-directory mapping assumed by Ruby - or a misuse of autoload_paths.

We have worked around this for now by removing lib/** from our autoload_paths and adding the necessary require statements to application.rb.

Answer 2

I looked into the rails source and there is a

if file_path && ! loaded.include?(File.expand_path(file_path)) # We found a matching file to load
  require_or_load file_path
  raise LoadError, "Expected #{file_path} to define #{qualified_name}" unless local_const_defined?(from_mod, const_name)
  return from_mod.const_get(const_name)
elsif ...

clause in load_missing_constant method. I may guess that as require_or_load is called before raise, this might be a reason that on the second call in your example there is no error...

It would be interesting to see a minimal example where two files with identical structure behave differently. In your place i would make a copy of the application and would keep removing parts from it while the inconsistent behavior is present to see the minimal inconsistent example.

P.S. I have submitted a similar question here: http://www.ruby-forum.com/topic/2376956

Answer 3

Was pointed in the correct direction by Brendan, check your autoload_paths. I had a similar error and this was the culprit for me in application.rb :

config.autoload_paths += Dir["#{Rails.root}/app/models/[a-z]*"]

My app was not happy once I started using modules for the models along with having sub-directories. I changed mine to autoload just the non-module directories :

config.autoload_paths += Dir["#{Rails.root}/app/models/aaaaaaaaa"]
config.autoload_paths += Dir["#{Rails.root}/app/models/bbbbbbbbb"]

Answer 4

I've suffered from the very same problem with namespaced classes being autoloaded incorrectly. This is the clue to solve it (from Brendan's answer):

Since our autoload_paths included both lib and lib/**, Rails pulled in all the files from all the subdirectories under lib. Unfortunately, it appears to ignore the implied namespace when loading from subdirectories. It expected foo/base.rb to define Base vs. Foo::Base. That seems to be the core flaw: load_missing_constant invokes search_for_file which returns any file whose name matches (e.g., in my example, foo/base.rb was returned as it matched base.rb).

And I've solved it by moving all my namespaced classes, named Events::Something, into app/components/events/* and creating a file app/components/events.rb with the following content:

%w(file1 file2 ...).each do |file|  
  require "events/#{file}"
end

And while this is a little bit manual, it works both for application, console and test modes.