will googlebot index my site?

Question

in my robots.txt file, I have the following line

User-agent: Googlebot-Mobile Disallow: /

User-agent:GoogleBot Disallow: /

Sitemap: http://mydomain.com/sitemapindex.xml

I know that if I put the first 4 lines , googlebot won't index the sites, but what if I put the last line Sitemap: http://mydomain.com/sitemapindex.xml, will googlebot be able to index the site?

Thanks,

Answer 1

I tested your robots.txt against my own domain (which has a sitemap entry for every page) and Googlebot and Googlebot-Mobile returned that they were Disallowed access.

Based on this - I would say the robots.txt file takes precedence over any sitemaps.

Plus, logically speaking - if you block the entire domain, the bot is disallowed access to the sitemap. The sitemap entry just tells crawlers where to find your sitemap - not their authorization to access it.

Even if you allowed the sitemap, I don't think bots would crawl your site - sitemaps are designed more for telling the bot how often to crawl your site, not what they are allowed to crawl.