Rails4 ActionController::InvalidAuthenticityToken error

Question

I have Rails4 application running in production, and my visitors run occasionally into ActionController::InvalidAuthenticityToken error, which I cant reproduce. I get 2-4 daily notifications from various forms, with no clear logic behind. The report I receive shows that authenticity_token submitted by the form is different from one kept in session. How is it possible? I managed to run into the problem myself few times, however it's impossible to reproduce, all of sudden authenticity_token for the form is different from the one stored in session and InvalidAuthenticityToken arises.
Any ideas where to start looking?

Example:

 Request:
-------------------------------

  * URL        : https://domain/signin
  * HTTP Method: POST
  * IP address : 113.96.xx.xx
  * Parameters : {"utf8"=>"✓", "authenticity_token"=>"MOh9JDE1AZ0CbIw/M33vfhjRShwzI6oqMhi8lk+n7OE=", "email"=>"xxxx@xxx", "password"=>"[FILTERED]", "commit"=>"Sign In", "controller"=>"clients", "action"=>"signin", "locale"=>"en"}

-------------------------------
Session:
-------------------------------

  * session id: [FILTERED]
  * data: {"_csrf_token"=>"QazCSVGeZlxEh83XTM+f5PkC/zopwCF96yV4duRats0="}

Update: Wanted to add that I'm serving the pages via two load balanced AWS EC2 instances, and store sessions in Redis ElastiCache instance

Answer 1

Any forms generated by Rails (i.e. with form_for and such, not with you putting <form> in the template) will have the anti-CSRF token added as a hidden field when it's necessary. If you wrote the form yourself and didn't include the CSRF hidden input, then Rails relies on the CSRF meta tag and JavaScript for things to work. So if you wrote your own form and you didn't include the hidden field and if the client's JavaScript doesn't work for whatever reason, you can get this error. Because the "client's JavaScript doesn't work for whatever reason" clause is a difficult one to detect and debug, I actually intentionally removed the CSRF meta tag on my site. That way if I forgot to include the hidden input, it'll break for everyone (fast fail), I'd find out about it immediately, and I could fix it. I would recommend you do this as well.

With that being said, I would recommend you look at the access log for these "visitors". Do you see anything odd?

• Do they access the page that has the form on it immediately before submitting it? If not, maybe they're bots or actual, real-deal CSRF attempts (that's what this is checking for, isn't it? :)).
• Did they load the form on one EC2 and end up submitting to another? If so, can you turn one EC2 off and see if the errors go away?
• Did they somehow lose their session? This could be your problem or theirs.