Menu
I have a client who I am trying to setup an SSL certificate for via SSL for Free, like I have done 100 times before.
I created the file structure under public_html:
.well-known > pki-validation > <uploaded verification file>
I then tried to download the certificate and got the following failure message:
Warning: Your verification URL is not returning the correct contents to our verification servers. The URL looks like it is blocking bots and which inadvertently blocks our servers from receiving the correct content. Contact your host, a professional developer or admin for further help with fixing it.
I assumed this was due to the robots.txt file, had a look and it included the following:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
So I changed it to temporarily allow all bots using:
User-agent: *
Disallow:
I then retried the process, uploading the new verification file, but still, I get the same warning message about the URL blocking the bots. Has anyone come across this before and know how to fix it?
I've also tried verification via a DNS .txt record and via FTP, both returning the same failure notices.
Hosting is through GoDaddy and they're using Linux cPanel. The website is made through WordPress.
Typing in example.com/robots.txt returns the correct robots.txt file.
Thank you for all and any help!
866
The best way to block bot traffic from accessing your website or app is to use a Web Application Firewall (WAF).
If you have to, always make sure that block the required bots and spiders from crawling your site. This will result in enhanced security, a better overall online reputation, and a much better site that will be there in the years to come.
These bots are sent by various third-party service providers you use. For example, if you use SEO tools like Ahrefs or SEMRush, they will use their bots to crawl your site to check your SEO performance (link profile, traffic volume, etc.). Performance measurement tools such as Pingdom also fall in this category.
I've now fixed this issue. It was indeed to do with the robots.txt file as I had initially thought. The reason it didn't initially work is the site was still cached with the old file, which was blocking bots throughout the site.
I added some code to my .htaccess file to force re-validation of the cache:
ExpiresActive On
ExpiresDefault A1
Header append Cache-Control must-revalidate
After doing this and with my new robots.txt file, I was able to install the new SSL certificate to the website which is now working! I hope this helps anyone having the same issue.
Just remember to remove the above code afterwards, as it will temporarily stop your site from caching.
175
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With