Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Robots.txt for multiple domains

Tags:

We have different domains for each language

  1. www.abc.com
  2. www.abc.se
  3. www.abc.de

And then we have different sitemap.xml for each site. In robots.txt, I want to add sitemap reference for each domain.

  1. Is it possible to have multiple sitemap references for each domain in single robots.txt?
  2. If there are multiple, which one does it pick?
like image 512
Sachin Surana Avatar asked Jul 07 '12 08:07

Sachin Surana


People also ask

Can you have multiple robots txt?

Beyond www and non-www, a site can have a robots. txt file sitting at the https version of a subdomain and then also at the http version of that subdomain. So, similar to what I explained above, there could be multiple robots. txt files with different instructions based on protocol.

Do subdomains need their own robots txt?

Each subdomain is generally treated as a separate site and requires their own robots. txt file.

How many robots txt file can be in the website?

Your site can have only one robots. txt file.

Does every website have a robots txt file?

The robots. txt file helps major search engines understand where they're allowed to go on your website. But, while the major search engines do support the robots.


2 Answers

I'm using the following solution in .htaccess after all domain redirects and www to non-www redirection.

# Rewrite URL for robots.txt
RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [L]

Create a new directory in your root called robots. Create a text file filled with the specific robots information for every domain.

  • /robots/abc.com.txt
  • /robots/abc.se.txt
  • /robots/abc.de.txt
like image 58
hans2103 Avatar answered Sep 28 '22 07:09

hans2103


The robots.txt can only inform the search engines of sitemaps for its own domain. So that one will be the only one it honors when it crawls that domain's robots.txt. If all three domains map to the same website and share a robots.txt then the search engines will effectively find each sitemap.

like image 39
John Conde Avatar answered Sep 28 '22 06:09

John Conde