Menu
I see significant traffic coming from this Bot when we send SMS to users. I am trying to get more details about this bot. Any pointers describing what it does, how important it is, and can we block it would be appreciated. The complete user agent is
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)
408
Google web crawlers also referred to as Google bots, Google robots, or Google spiders are digital bots that crawl across the world wide web (www) to discover and index web pages for search engines such as Google, Bing, etc. Google doesn’t know what sites exist on the internet.
The goal of such a bot is to learn what (almost) every webpage on the web is about, so that the information can be retrieved when it's needed. They're called "web crawlers" because crawling is the technical term for automatically accessing a website and obtaining data via a software program. These bots are almost always operated by search engines.
The more Googlebot visits your site, the faster your site’s content updates will appear in Google’s search results. Consequently, it’s of the utmost importance to allow Googlebot to crawl your website without blocking or disturbing it.
There are two methods for verifying Google's crawlers: Manually: For one-off lookups, use command line tools. This method is sufficient for most use cases. Automatically: For large scale lookups, use an automatic solution to match a crawler's IP address against the list of published Googlebot IP addresses.
Google's SMS app on Android uses this user agent when it crawls the shared URL to generate a preview inside the chat. This is usually a good thing because your customers will see the preview image and page title, rather than just a link. So I would not recommend blocking it.
57
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
