Googlebot receiving missing template error for an existing template

Question

In the last couple of days, we have started to receive a missing template error when the google bot attempts to access our main home page (welcome/index). I have been staring at this for a couple of hours and know that I am just missing something simple.

A ActionView::MissingTemplate occurred in welcome#index:
Missing template welcome/index with {:handlers=>[:erb, :rjs, :builder, :rhtml, :rxml, :haml], :formats=>["*/*;q=0.9"], :locale=>[:en, :en]}

But the template does exist (index.html.haml). If it didn't no one could access our home page.

Here is some additional environment information:

* REMOTE_ADDR                               : 66.249.72.139
* REMOTE_PORT                               : 56883
* REQUEST_METHOD                            : GET
* REQUEST_URI                               : /

* Parameters: {"controller"=>"welcome", "action"=>"index"}

Any insights you have would be greatly appreciated.

Answer 1

These errors are coming from the way GoogleBot formats its HTTP_ACCEPT header. While valid (see W3 reference), it adds a q=0.6 (last figure may change) which is used as a separator. Since there is no other media type specified, this q=0.6 is not necessary and I assume this is why Rails doesn't treat the header correctly.

(It seems to depend on Rails version. On Rails 3.0.12, it raises a MissingTemplate exception.)

Adding the following code from a previous answer to the concerned controller is not sufficient: it responds with an error 406.

respond_to do |format|
  format.html
end

To make this work under Rails 3.0.12 and have something returned to the GoogleBot (better than a 406 error), you need to add this code which sets the request's format to html as soon a */*;q=0.6-like HTTP_ACCEPT is detected (Rails load the header value into request.format).

# If the request 'HTTP_ACCEPT' header indicates a '*/*;q=0.6' format,
# we set the format to :html.
# This is necessary for GoogleBot which perform its requests with '*/*;q=0.6'
# or similar HTTP_ACCEPT headers.
if request.format.to_s =~ %r%\*\/\*%
  request.format = :html
end

respond_to do |format|
  format.html
end

While working, this solution needs the code to be added to any controller action you want to be indexed by the GoogleBot, what is really not DRY!

To fix this issue once for all, I implemented a small Rack middleware which does even better: it checks the request's HTTP_ACCEPT header, and will replace any header matching */*;q=0.6 (the figures can vary) by the common */*. This is even better because since the q=0.6 has no meaning if it is not followed by another media type, this change of the header doesn't change its meaning. We don't force Rails into any given format, we just tell it any will do in a way it can understand.

You can find the middleware, the loading initializer and an integration test in this gist.

Gem version here: https://github.com/ouvrages/rails_fix_google_bot_accept

Answer 2

I am also getting the same, I did some investigation and came to the conclusion it is a 'bug' in Rails. */*;q=0.9 is the value of the HTTP accept parameter. I'm not exactly sure what is going on, but in Rails 3.0 this works. In Rails 3.1 it returns a 500 response, and in Rails 3.2 it returns a 406 response.

Update:

There is an open bug regarding this issue. One workaround is to set this new option in Rails 3.1:

config.action_dispatch.ignore_accept_header = true

However... if you serve any pages other than HTML you'll need to rely on the extension to denote the type (e.g. /users/1.json) instead of accept headers.

Answer 3

The solution to the problem is to specify the format in your action.

Up until now, I had simply had the following in my index action

def index

end

Once I inserted a respond_to block

def index
  respond_to do |format|
    format.html
  end
end

I stopped getting the missing template errors.