Menu
I have a site that I am using a Google Custom Search Engine on. I want Google CSE to crawl my site but I want it to stay out of the results of a regular Google search. I put this in my robots.txt file hoping that google CSE bots would ignore it since I specified the pages I wanted Google CSE to crawl in the settings
User-agent: *
Disallow: /
I guess the Google CSE bots also have to obey robots.txt. So is there a way to get my page to stay out of search engine searches but for Google CSE to still be able to index it? TIA!
853
Using the robots. txt file you can prevent search engines from accessing certain parts of your website , prevent duplicate content and give search engines helpful tips on how they can crawl your website more efficiently .
In practice, robots. txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents.
Prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag: <meta name="googlebot" content="noindex, nofollow">.
There is no solution to this question that fits with what you would like. I too have the same situation where I need custom search only. Unfortunately Google's list of crawlers doesn't show a specific bot for google custom search. Blocking Googlebot will both kill native search and custom search.
69
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
