Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is Yahoo Indexing Bot considered as "evil"?

After reading and commenting on this question PHP Library for Keeping your site index by Google, Bing, etc, I was curious to look at StackOverFlow's sitemap. This returned a 404 error which I am guessing is just a protected page by determining if your are a Index Bot or simply doesnt exists.

This then lead me to look at the robots.txt for StackOverFlow. I was surprised to see the comment "Yahoo bot is evil" along with a couple other Indexing bots (Spinn3r and KSCrawler) . I am unfamilular with Spinn3r and KSCrawler but my question is, why are these bots (particular Yahoo) considered as evil? Surely any and all indexing of any Search Engine is a good thing?

like image 676
bigstylee Avatar asked Apr 01 '10 18:04

bigstylee


1 Answers

These are very aggressive bots that do not throttle requests and can bring a site down.

See this forum thread for an example for the yahoo bot.

like image 168
Oded Avatar answered Sep 28 '22 22:09

Oded