Robots.txt file on Rails heroku app not updating

Question

I've got a rails app hosted on Heroku for which I'm trying to update the robot.txt file.

The local file, which is at /public/robots.txt, reads:

User-agent: *
Disallow: /admin/

However when I deploy the app to Heroku, the robots file is seemingly not updated. The remote version reads:

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the     robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-agent: *
Disallow: /

The live url is at http://www.cowboypicks.com/robots.txt.

Running curl -I http://www.cowboypicks.com/robots.txt yields:

HTTP/1.1 200 OK
Age: 2317078
Cache-Control: public, max-age=2592000
Content-length: 200
Content-Type: text/plain
Date: Wed, 30 Apr 2014 17:01:43 GMT
Last-Modified: Thu, 03 Apr 2014 14:21:08 GMT
Status: 200 OK
X-Content-Digest: 10c5b29b9aa0c6be63a410671662a29a796bc772
X-Rack-Cache: fresh
Connection: keep-alive

Indicating the file hasn't been updated since 3 April, however it has been updated today (30 April). Even stranger, when I run heroku run bash followed by cat public/robots.txt I get:

User-agent: *
Disallow: /admin/

Indicating the file is being updated on Heroku but it's showing an older (I assume cached) version for some reason. There is some caching on the app using dalli/memcache, but I wouldn't have thought that would affect static files? Any ideas on how I could further debug the problem?

Answer 1

It turned out that Dalli had indeed been caching the robots.txt file in production. The expiry date was set in production.rb with the line:

 config.static_cache_control = "public, max-age=2592000"

Running the following from the rails console flushed the cache and sorted the problem:

Rails.cache.dalli.flush_all