sunspot_rails unable to index due to "404 Not Found" error

Question

I'm trying to install Sunspot in a small Rails app, exactly following the gem setup instructions, but every time I run into RSolr::Error::Http: RSolr::Error::Http - 404 Not Found errors when I try to index data. I can reproduce this with a fresh app; here are the exact steps I follow:

Create a fresh Rails 4.2.5 app:

$ rails new test_sunspot
$ cd test_sunspot/
$ spring stop # spring can cause `generate` commands to hang
$ rails g model Thing title:string
$ rake db:migrate
$ rails c
  > Thing.create!(title: "Cats")
  > Thing.create!(title: "Pizza")
  > exit

Add a Sunspot index to the model:

class Thing < ActiveRecord::Base
  searchable do
    text :title
  end
end

Add Sunspot to Gemfile:

...
gem 'sunspot_rails', '2.2.2'
gem 'sunspot_solr',  '2.2.2'  
...

Install, start, and reindex Sunspot:

$ bundle install
$ rails g sunspot_rails:install # default sunspot.yml is not changed
$ ps aux | grep solr # confirm that no Solr services are running
$ bundle exec rake sunspot:solr:start # generates solr/ dir; no errors
$ bundle exec rake sunspot:solr:reindex

This reindex command yields the following output. When I go into the Rails console and tried to create a new Thing object, it triggers the same error (because Sunspot attempts to update the index):

Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile
rake aborted!
RSolr::Error::Http: RSolr::Error::Http - 404 Not Found
Error:     Not Found

URI: http://localhost:8982/solr/development/update?wt=ruby
Request Headers: {"Content-Type"=>"text/xml"}
Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete><query>type:Thing</query></delete>"

Backtrace: /Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:284:in `adapt_response'
/Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:190:in `execute'
/Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:176:in `send_and_receive'
# ...lots of backtrace omitted...
/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_rails-2.2.2/lib/sunspot/rails/tasks.rb:19:in `block (2 levels) in <top (required)>'
/Users/topher/.rvm/gems/ruby-2.2.0/bin/ruby_executable_hooks:15:in `eval'
/Users/topher/.rvm/gems/ruby-2.2.0/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => sunspot:solr:reindex => sunspot:reindex
(See full trace by running task with --trace)

The output of ps aux | grep solr (after starting Solr): Note that the PID mentioned in solr/pids/development/sunspot-solr-development.pid is 62449, which matches the third line:

topher          62617   0.0  0.0  2432772    520 s002  R+    3:00PM   0:00.00 grep solr
topher          62484   0.0  1.3  3274624 105756   ??  S     2:57PM   0:03.65 /usr/bin/java -server -Xss256k -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_solr-2.2.2/solr/server/logs/solr_gc.log -DSTOP.PORT=7982 -DSTOP.KEY=solrrocks -Djetty.port=8982 -Dsolr.solr.home=/Users/topher/Sites/john_kole/test_sunspot/solr -Dsolr.install.dir=/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_solr-2.2.2/solr -Duser.timezone=UTC -Djava.net.preferIPv4Stack=true -jar start.jar
topher          62449   0.0  0.0  2444632   1304   ??  Ss    2:57PM   0:00.04 bash ./solr start -f -p 8982 -s /Users/topher/Sites/john_kole/test_sunspot/solr

Other details:

• I'm on Mac OSX Yosemite
• I've uninstalled and reinstalled the relevant gems, tried downgrading to Sunspot 1.3.0, and even ran gem pristine --all, with no change in outcome
• EDIT: I have read through the other similar tickets on Sunspot/Solr 404 Not Found errors. It looks like the solutions in those cases amounted to "resetting" the Solr config and aren't relevant to a fresh (dev environment) project: no prior Solr instances are running before I run rake sunspot:solr:start; this is a completely fresh Rails project so the solr/ directory was just newly generated anyway; this is in development, not production; and adding solr_home: solr to sunspot.yml or updating path: /solr/default have no effect on the 404 outcome.

Questions:

• Any idea why this is happening?
• Could this breakage be due to something mis-installed in my OSX environment? What should I look at?
• Can you get Sunspot working properly on a fresh install of current Rails, following these same steps?

Thanks in advance!

Answer 1

Remove the sunspot_solr gem and the install solr on your machine by the following commands

brew install solr

if it doesn't work run brew update first to fetch the latest solr links

brew will install all missing dependencies automatically.

To start solr:

solr start

Answer 2

TL;DR: You have done everything right, but the user installing the gems is not the same as the one running solr (aka permissions issue)

I tried to reproduce your case and succeeded, which felt very wrong. The issue is that installing the gem sunspot_solr ships an embedded Java webserver Jetty. I managed to trace the cause of the problem to the fact that this webserver is not run as the person who installed it.

This can be verified by installing gems per:

$ touch Gemfile # to mark it as edited
$ bundle install --path vendor/bundle # to install as user who runs the server later

instead of the usual

$ gem install ... bundle install

as another (possibly root) user.
Now you can succeed performing all your steps as mentioned in your post.