Why does Googlebot crawl for /mobile/* and /m/* pages that are not referenced anywhere?

Question

Since the end of may, I have a lot of new 404 errors in the Smartphone Crawl Errors page in Webmaster Tools / Google search console. All of them starts with /m/ or /mobile/, none of which are existing nor linked to anywhere on the site.

For example, I have a 404 error for http://www.example.com/mobile/foo-bar/ and http://www.example.com/m/foo-bar pages. According to the Search Console, those page are linked in the existing page http://www.example.com/foo-bar/, but they are not.

Is Googlebot deciding on its own to look for a mobile version of every page ? Can I disable this behavior ? Is this because my site is not mobile-friendly yet (a problem for which I received another warning message from Google).

Answer 1

As @Jonny 5 mentioned in a comment, this seems to be happening as a result of Google guessing that you may have a mobile version of your site in the /m and/or /mobile directories. From what I have read, they will only try those directories if they decided that the pages they initially indexed were not mobile-friendly/responsive. More info on this behavior can be found in these Google Product Forum threads:

• https://productforums.google.com/forum/#!topic/webmasters/k3TFeCkFE0Q
• https://productforums.google.com/forum/#!topic/webmasters/56CNFxZBFwE

Another helpful comment came from @user29671, who pointed out that your website does in fact have some URLs with /m and /mobile indexed. I found that the same was true for my website, so this behavior may also be limited to sites that Google has (for whatever reason) indexed a /m and/or /mobile URL for. To test if this is true for your site, go to the following URLs and replace example.com with your website's domain:

• https://www.google.com/search?q=site:example.com/m/&filter=0
• https://www.google.com/search?q=site:example.com/mobile/&filter=0

As far as preventing this goes, your best bet is either creating a mobile-friendly version of your site or redirecting /m and /mobile pages back to the originals.