Menu
I am having the hardest time determining the cause of our Asp.net site not getting indexed in search engines - the entire site. When I use google's "Fetch As Googlebot" tool, it throws the below error. I made sure my site works with sessions disabled (ie. setting sessionMode="Off" in web.config), I've googled all over the web, and still no luck.
Here's the error I am getting from googlebot:
HTTP/1.1 302 Found
Date: Thu, 02 Dec 2010 23:05:49 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: /ErrorPage.aspx?aspxerrorpath=/Default.aspx
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 168
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="%2fErrorPage.aspx%3faspxerrorpath%3d%2fDefault.aspx">here</a>.</h2>
</body></html>
783
I found the answer myself.
Make sure to check Request.UserLanguages != null before using it. Also, make sure the CurrentCulture is set to a valid default value. The reason for both these checks is because bots don't use Request.UserLanguages - it's always null. Browsers do use Request.UserLanguages. To restate it in other words: Don't set the CurrentCulture if Request.UserLanguages is null.
Here's the problem:
All the aspx pages in my site inherit from a custom base class that inherits from System.Web.UI.Page. This isn't a problem until you override the OnLoad() or init events with code that throws an exception for bots only. I had this line of code in my OnLoad() event:
Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture(Request.UserLanguages[0]);
This code works great when accessing aspx pages from a browser. Duh! who isn't going to use a browser...? answer: googlebot and all other bots.
How to check if googlebot has a problem indexing your site:
171
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
