How can I configure my site to allow crawling from well known robots like google, bing, yahoo, alexa etc. and stop other harmful spammers, robots
should i block particular IP? please discuss any pros, cons Anything to be done in web.config or IIS?
Can I do it server wide If i have vps with root access?
Thanks.
To prevent specific articles on your site from being indexed by all robots, use the following meta tag: <meta name="robots" content="noindex, nofollow">. To prevent robots from crawling images on a specific article, use the following meta tag: <meta name="robots" content="noimageindex">.
A robots. txt file is a text file that specifies the rules for any bots accessing the hosted website or application. These rules define which pages the bots can crawl, and which links they can follow.
One option to reduce server load from bots, spiders, and other crawlers is to create a robots. txt file at the root of your website. This tells search engines what content on your site they should and should not index.
I'd recommend that you take a look the answer I posted to a similar question: How to identify web-crawler?
Robots.txt
The robots.txt is useful for polite bots, but spammers are generally not polite so they tend to ignore the robots.txt; it's great if you have robots.txt since it can help the polite bots. However, be careful not to block the wrong path as it can block the good bots from crawling content that you actually want them to crawl.
User-Agent
Blocking by user-agent is not fool-proof either, because spammers often impersonate browsers and other popular user agents (such as the Google bots). As a matter of fact, spoofing the user agent is one of the easiest thing that a spammer can do.
Bot Traps
This is probably the best way protect yourself from bots that are not polite and that don't correctly identify themselves with the User-Agent. There are at least two types of traps:
A hidden link is one which is not visible to a person, such as an anchor tag with no text: <a href="http://www.mysite.com/path/to/bot/trap"></a>
. Alternately, you can have text in the anchor tag, but you can make the font really small and change the text color to match the background color so that humans can't see the link. The hidden link trap can catch any non-human bot, so I'd recommend that you combine it with the robots.txt trap so that you only catch bad bots.
Verifying Bots
The above steps will probably help you get rid of 99.9% of the spammers, but there might be a handful of bad bots who impersonate a popular bot (such as Googlebot) AND abide by your robots.txt; those bots can eat up the number of requests you've allocated for Googlebot and may cause you to temporarily disallow Google from crawling your website. In that case you have one more option and that's to verify the identity of the bot. Most major crawlers (that you'd want to be crawled by) have a way that you can identify their bots, here is Google's recommendation for verifying their bot: http://googlewebmastercentral.blogspot.com/2006/09/how-to-verify-googlebot.html
Any bot that impersonates another major bot and fails verification can be blocked by IP. That should probably get you closer to preventing 99.99% of the bad bots from crawling your site.
Blocking by IP can be useful, but the method that I use is blocking by user-agent, that way you can trap many different IPs using apps that you don't want, especially site grabbers. I won't provide our list as you need to concentrate on those that affect you. For our use we have identified more than 130 applications that are not web browsers and not search engines that we don't want accessing our web. But you can start with a web search on user-agents for site grabbing.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With