URL Blocking Bots

Question

I have a client who I am trying to setup an SSL certificate for via SSL for Free, like I have done 100 times before.

I created the file structure under public_html:

• .well-known > pki-validation > <uploaded verification file>

I then tried to download the certificate and got the following failure message:

Warning: Your verification URL is not returning the correct contents to our verification servers. The URL looks like it is blocking bots and which inadvertently blocks our servers from receiving the correct content. Contact your host, a professional developer or admin for further help with fixing it.

I assumed this was due to the robots.txt file, had a look and it included the following:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

So I changed it to temporarily allow all bots using:

User-agent: * 
Disallow:

I then retried the process, uploading the new verification file, but still, I get the same warning message about the URL blocking the bots. Has anyone come across this before and know how to fix it?

I've also tried verification via a DNS .txt record and via FTP, both returning the same failure notices.

Hosting is through GoDaddy and they're using Linux cPanel. The website is made through WordPress.

Typing in example.com/robots.txt returns the correct robots.txt file.

Thank you for all and any help!

Answer 1

I've now fixed this issue. It was indeed to do with the robots.txt file as I had initially thought. The reason it didn't initially work is the site was still cached with the old file, which was blocking bots throughout the site.

I added some code to my .htaccess file to force re-validation of the cache:

ExpiresActive On
ExpiresDefault A1
Header append Cache-Control must-revalidate

After doing this and with my new robots.txt file, I was able to install the new SSL certificate to the website which is now working! I hope this helps anyone having the same issue.

Just remember to remove the above code afterwards, as it will temporarily stop your site from caching.