Menu
We've had an accidental situation where our Wordpress site had the robots.txt set to disallow crawling for about 7 days. I'm now trying to play clean up and Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.txt." AFTER I"ve adjusted the robots.txt and allowed crawling. There is no reason why the URLS would still be blocked and when I visit the examples they appear OK.
Robots.txt URL: http://bit.ly/1u2Qlbx
Sitemap URL: http://bit.ly/1BfkSmx
My URLs where the robots.txt is blocking according to Webmaster Tools: http://bit.ly/1uLBRea OR http://bit.ly/1CsrHnr
371
In order to fix this, you will need to make sure that your website's robots. txt file is configured correctly. You can use the robots. txt testing tool from Google to check your file and make sure that there are no directives that are blocking Googlebot from accessing your site.
Blocked sitemap URLs are typically caused by web developers improperly configuring their robots. txt file. Whenever you're disallowing anything you need to ensure that you know what you're doing otherwise, this warning will appear and the web crawlers may no longer be able to crawl your site.
And just like robots. txt, an XML sitemap is a must-have. It's not only important to make sure search engine bots can discover all of your pages, but also to help them understand the importance of your pages. You can check your sitemap has been setup correctly by running a Free SEO Audit.
Don't worry, if you enabled the URLs again, Google's web crawlers will notice and crawl your pages again. The message will disappear from GWT within days.
Eventually, you may test your robots.txt with GWT's corresponding feature.
102
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
