Disallow or Noindex on Subdomain with robots.txt

Question

I have dev.example.com and www.example.com hosted on different subdomains. I want crawlers to drop all records of the dev subdomain but keep them on www. I am using git to store the code for both, so ideally I'd like both sites to use the same robots.txt file.

Is it possible to use one robots.txt file and have it exclude crawlers from the dev subdomain?

Answer 1

You could use Apache rewrite logic to serve a different robots.txt on the development domain:

<IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteCond %{HTTP_HOST} ^dev\.qrcodecity\.com$
    RewriteRule ^robots\.txt$ robots-dev.txt
</IfModule>

And then create a separate robots-dev.txt:

User-agent: *
Disallow: /