Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to set up a robot.txt which only allows the default page of a site

Tags:

Say I have a site on http://example.com. I would really like allowing bots to see the home page, but any other page need to blocked as it is pointless to spider. In other words

http://example.com & http://example.com/ should be allowed, but http://example.com/anything and http://example.com/someendpoint.aspx should be blocked.

Further it would be great if I can allow certain query strings to passthrough to the home page: http://example.com?okparam=true

but not http://example.com?anythingbutokparam=true

like image 400
Boaz Avatar asked Sep 04 '08 09:09

Boaz


2 Answers

So after some research, here is what I found - a solution acceptable by the major search providers: google , yahoo & msn (I could on find a validator here) :

User-Agent: * Disallow: /* Allow: /?okparam= Allow: /$ 

The trick is using the $ to mark the end of URL.

like image 55
Boaz Avatar answered Sep 22 '22 08:09

Boaz


Google's Webmaster Tools report that disallow always takes precedence over allow, so there's no easy way of doing this in a robots.txt file.

You could accomplish this by puting a noindex,nofollow META tag in the HTML every page but the home page.

like image 30
ceejayoz Avatar answered Sep 24 '22 08:09

ceejayoz