Menu
I am implementing a simplistic filter on how much of my site unregistered users can acces. Naturally, I want to give SEO bots free reign/access to most of the site.
I know this is simplistic. But its not worth doing anything more complicated. I need to compile a list of names of user agents I will allow, for this, I need a list of the names of the bots, starting with Googlebot (I don't even know if that is the official spelling of Gogles web crawling bot).
Anyone I would like a link to a definite resource that gives the names of the SEO indexing bots?. I tried http://www.user-agents.org/ but the granularity is not fine enough - it appears to list every user agent ever created!
389
A user agent string is a short “string” of text that identifies the browser to the web server — every browser has a unique one!
Browse our database of 219.4 million User Agents - WhatIsMyBrowser.com.
Alternatively, you can identify Googlebot by IP address by matching the crawler's IP address to the list of Googlebot IP addresses. For other Google IP addresses from where your site may be accessed (for example, by user request or Apps Scripts), match the accessing IP address against the list of Google IP addresses.
Try this list: http://www.useragentstring.com/pages/useragentstring.php?typ=Crawler
Although the combination of Google, Yahoo, Bing, Baidu, Ask, and AOL represent virtually 100% of the search engine market. So I would recommend adding the crawler user agents for just those to your filter and you really don't need to worry about the rest.
170
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
