'rake spec' doesn't run with custom RAILS_ENV but 'rspec' and 'bundle exec rspec' do

Question

So I'm having a bit of a bizarre issue. I have a custom rails environment setup for my CI server which is running linux. The environment gets loaded properly, and the tests DO run on the CI server, but only if I run them with bundle exec rspec instead of bundle exec rake spec or bundle exec rake.

When the tests don't run, they still return an exit code of 0 and the CI server assumes the build was successful, even if theoretically the build may be broken.

Identical behavior occurs on my machine, which is running OS X.

Here's a console session with all the different test cases to better illustrate what's happening.

No RAILS_ENV specified

~/myapp $ rake
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 4.34 seconds (files took 3.1 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ rake spec
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.8 seconds (files took 3.36 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ rspec
...............................................

Finished in 3.87 seconds (files took 2.98 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ bundle exec rake
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.9 seconds (files took 3.03 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ bundle exec rake spec
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.64 seconds (files took 2.97 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ bundle exec rspec
...............................................

Finished in 3.75 seconds (files took 2.95 seconds to load)
47 examples, 0 failures

Everything is working as expected

With a standard RAILS_ENV

Results are identical with test, development, or production

~/myapp $ RAILS_ENV=test rake
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.86 seconds (files took 3.07 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=test rake spec
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.9 seconds (files took 3.02 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=test rspec    
...............................................

Finished in 3.82 seconds (files took 2.98 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=test bundle exec rake
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.76 seconds (files took 2.91 seconds to load)
47 examples, 0 failures
# ------
~/myapp $ RAILS_ENV=test bundle exec rake spec
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.83 seconds (files took 2.99 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=test bundle exec rspec    
...............................................

Finished in 3.83 seconds (files took 3.11 seconds to load)
47 examples, 0 failures

Once again everything is fine and dandy.

With a custom RAILS_ENV

~/myapp $ RAILS_ENV=ci rake
/Users/io/.rvm/rubies/ruby-2.2.0/bin/ruby -I/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/lib:/Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-support-3.3.0/lib /Users/io/.rvm/gems/ruby-2.2.0/gems/rspec-core-3.3.2/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
...............................................

Finished in 3.73 seconds (files took 3.03 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=ci rake spec
# No output, just a brief pause and back to shell prompt
~/myapp $ RAILS_ENV=ci rspec    
...............................................

Finished in 7.82 seconds (files took 2.96 seconds to load)
47 examples, 0 failures
# -----
~/myapp $ RAILS_ENV=ci bundle exec rake
# No output, just a brief pause and back to shell prompt
~/myapp $ RAILS_ENV=ci bundle exec rake spec
# No output, just a brief pause and back to shell prompt
~/myapp $ RAILS_ENV=ci bundle exec rspec    
...............................................

Finished in 7.77 seconds (files took 2.8 seconds to load)
47 examples, 0 failures

Due to the nature of the CI server's setup, I have to run the tests within bundle exec. But what I find mind boggling is that on my machine RAILS_ENV=ci rake works flawlessly, while RAILS_ENV=ci rake spec doesn't run.

But when I wrap them in a bundle exec neither rake nor rake spec run in RAILS_ENV=ci but bundle exec rspec works fine.

Can someone explain what is going on here? I can't find a way for this to make any logical sense. Have I stumbled upon a bug in Rails' rake tasks?

EDIT: In response to haradwaith's answer: You bring up some really good points, and the first point is spot on (it runs the rspec executable directly), but I can't say your answer adequately explains what's happening.

1. This behavior is not specific to my machine, the behavior is identical on a clean ruby:2.2 Docker container which runs bundle install before every test. As it loads a clean container each time, there is no way for "old" versions of gems to creep in.

2. To further replicate the "clean slate" of the Docker container, I just did a test on my machine with an empty gemset to rule out gem version conflicts and got identical results: http://pastebin.com/9u3MJaFw

3. My Gemfile does indeed have no mention of a :ci group. If the rspec-rails gem isn't being loaded, then RAILS_ENV=ci rake with no arguments wouldn't run Rspec as the default Rake task, but it clearly does. We can also see that it has different behavior depending on whether or not its being ran with bundle exec. I don't believe that running with bundle exec would "unload" rspec-rails if running it without bundler somehow manages to load it automatically.

4. If rspec and/or rspec-rails weren't being loaded, the rake would fail with an exit code other than 0, and spit out something like:

   ~/not-myapp $ touch Rakefile

   ~/not-myapp $ rake spec

   rake aborted!

   Don't know how to build task 'spec'

   (See full trace by running task with --trace)

Answer 1

Several points:

rspec

rspec and bundle exec rspec always run your tests because it uses directly the rspec executable.

rake

The gem rspec-rails defines some rake tasks, including the spec task. However, rake will only be able to run this task if the rspec gem has been loaded in the used environment. If you have something like this in your Gemfile:

group :test, :development do
  gem 'rspec-rails', '~> 3.0'
end

The spec task command will only work in the test and development environment, and not with RAILS_ENV=ci.

bundle

It is recommended to always use bundle exec to run a ruby executable. From the Bundler website:

In some cases, running executables without bundle exec may work, if the executable happens to be installed in your system and does not pull in any gems that conflict with your bundle.

However, this is unreliable and is the source of considerable pain. Even if it looks like it works, it may not work in the future or on another machine.

Without bundle exec, the executable may have unexpected or inconsistent behavior, by mixing other gems or gem versions already installed on your machine.

In your case, the fact that RAILS_ENV=ci rake works and RAILS_ENV=ci rake spec doesn't could be caused by some conflict with an older version of rspec-spec, or by some other gem defining other default rake task.

In any case, this behavior is nor a bug and is probably specific to your machine. I'd recommend using exclusively bundle exec rspec to run your tests.

---

EDIT: In response to your remarks:

4 - The fact that rake spec doesn't crash even if rspec-rails is not loaded is a tricky part. When you try to run a task example, rake first searches for the task example in the loaded rakefiles. If it doesn't find it, it will check if the file example exists in your project. If it does, it will try to execute so task from this file.

In your case, the file spec exists in your project, and is a directory. So when you do rake spec, if the task spec doesn't exists, it will load your spec file, and do nothing before there is no task to run in it.

That's why rake blublu will fail, but rake spec or rake app won't.