Robots.txt for umbraco

Question

I wonder if there is a default Robots.txt for Umbraco (7.5) websites. Right now, I created one with these items. Shall I add anything else?

User-Agent: *
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /css
Disallow: /Errorpages/
Disallow: /scripts/
Disallow: /Config/

Answer 1

You should consider removing /css and /scripts from the list, actually. I believe Google finds it friendlier/better for those to be available.

Otherwise, maybe put /install/ in there as well, just as a precaution.

Answer 2

Regarding CSS & JSS check this: https://yoast.com/dont-block-your-css-and-js-files/. I decided to cut them off lately.

Mine, always used with Umbraco instances, robots.txt file looks like this:

# robots.txt for Umbraco
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /data/
Disallow: /install/
Disallow: /macroScripts/
Disallow: /masterpages/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/
User-agent: Libwww-perl
Disallow: /
Sitemap: http://{HTTP_HOST}/sitemap