When digging through a Google robots.txt file I noticed a line that I was not familiar with.
What does the below code mean in the context of a robots.txt file?
Allow: /$
Does the '$' change the meaning any from simply saying
Allow: /
According to SearchEngineLand..
You can match the end of the string using the dollar sign ($). For example, to block URLs that end with .asp:
User-agent: Googlebot
Disallow: /*.asp$
Looks like it acts a bit like a regular expression.
Some robots understand pattern matching. $
is used in regular expressions to mark end of line. So this rule should Allow /
but not /foo
.
I have no source for robots.txt particular though.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With