Rails robots.txt folders

Question

I'm about to launch a Rails app and as the last task, I wan't to set the robots.txt file. I couldn't find information about how the paths should be written properly for a Rails app.

Is the starting path always the root path from the Rails app or the app folder? How would I then disallow e.g. the img folder?

Do I have to write the paths as I see it in the app folder, or like how the paths look like on the site online, e.g. http://example.com/admin ?

Answer 1

you have to put your robots.txt in /public folder

and it will look like in below

The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

for more information please follow below links

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

and

http://www.robotstxt.org/robotstxt.html