Identifying hostile web crawlers

Question

I am wondering if there are any techniques to identify a web crawler that collects information for illegal use. Plainly speaking, data theft to create carbon copies of a site.

Ideally, this system would detect a crawling pattern from an unknown source (if not on the list with the Google crawler, etc), and send bogus information to the scraping crawler.

• If, as a defender, I detect an unknown crawler that hits the site at regular intervals, the attacker will randomize the intervals.
• If, as a defender, I detect the same agent/IP, the attacker will randomize the agent.

And this is where I get lost - if an attacker randomizes the intervals and the agent, how would I not discriminate against proxies and machines hitting the site from the same network?

I am thinking of checking the suspect agent with javascript and cookie support. If the bogey can't do either consistently, then it's a bad guy.

What else can I do? Are there any algorithms, or even systems designed for quick on-the-fly analysis of historical data?

Answer 1

My solution would be to make a trap. Put some pages on your site where access are banned by robots.txt. Make a link on you page, but hide it with CSS, then ip ban anybody who goes to that page.

This will force the offender to obey robots.txt, which means that you can put important information or services permanently away from him, which will make his carbon-copy clone useless.

Answer 2

Don't try and recognize by IP and timing or intervals--use the data you send to the crawler to trace them.

Create a whitelist of known good crawlers--you'll serve them your content normally. For the rest, serve pages with an extra bit of unique content that only you will know how to look for. Use that signature to later identify who has been copying your content and block them.