Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Multiple User Agents in Robots.txt

Tags:

seo

robots.txt

In robots.txt file I have following sections

User-Agent: Bot1
Disallow: /A

User-Agent: Bot2
Disallow: /B

User-Agent: *
Disallow: /C

Will statement Disallow:c be visible to Bot1 & Bot2 ?

like image 294
GoodSp33d Avatar asked Feb 05 '13 14:02

GoodSp33d


People also ask

What does user agent * Disallow mean?

user-agent : identifies which crawler the rules apply to. allow : a URL path that may be crawled. disallow : a URL path that may not be crawled. sitemap : the complete URL of a sitemap.

Is robots.txt good for SEO?

You can use it to prevent search engines from crawling specific parts of your website and to give search engines helpful tips on how they can best crawl your website. The robots. txt file plays a big role in SEO.

Is robots.txt deprecated?

Google announced back in 2019 that the robots. txt to block indexing would no longer be honored.

Should robots.txt be indexed?

Rules in the robots. txt file are case-sensitive. In this case, it is recommended to make sure that only one version of the URL is indexed using canonicalization methods.


1 Answers

tl;dr: No, Bot1 and Bot2 will happily crawl paths starting with C.

Each bot only ever complies to at most a single record (block).

Original spec

In the original specification it says:

If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records.

Expired RFC draft

The original spec, including some additions (like Allow) became a draft for RFC, but never got accepted/published. In 3.2.1 The User-agent line it says:

The robot must obey the first record in /robots.txt that contains a User-Agent line whose value contains the name token of the robot as a substring. The name comparisons are case-insensitive. If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited.

So it confirms the interpretation of the original spec.

Implementations

Google, for example, gives an example that seems to follow the spec:

Each section in the robots.txt file is separate and does not build upon previous sections. For example:

User-agent: *
Disallow: /folder1/

User-Agent: Googlebot
Disallow: /folder2/

In this example only the URLs matching /folder2/ would be disallowed for Googlebot.

like image 189
unor Avatar answered Sep 18 '22 14:09

unor