Is it acceptable to have unhandled exceptions in ASP.NET web applications?

Question

I'm working with a third party vendor at the moment who has supplied an ASP.Net web application. The web app generates around 200 unhandled exceptions per day, which end up as emails in my in-box. Upon investigation it turns out that most of these errors are triggered by the GoogleBot web crawler indexing the site and triggering access to another third party web service, which is rate-limiting the requests. When a request limit is exceeded, the third party web service refuses the request, this results in an unhandled exception in the web server and an HTTP/500 status code. The exception looks like this:

Exception: Exception of type 'System.Web.HttpUnhandledException' was thrown., Stack Trace:    at System.Web.UI.Page.HandleError(Exception e)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.views_products_detail_aspx.ProcessRequest(HttpContext context)
   at System.Web.Mvc.ViewPage.RenderView(ViewContext viewContext)
   at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
   at System.Web.Mvc.ControllerActionInvoker.c__DisplayClass11.b__e()
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)

The web app developer seems unwilling to handle these errors for reasons I don't really understand. Their approach is to throttle the GoogleBot until the errors stop happening (Google indexes quite aggressivley, generating around 5,000 hits per day). While I accept that throttling the GoogleBot would work, it seems like a cop-out to me. I've always considered unhandled exceptions to be bugs. Shouldn't the web app handle these errors? It is ever acceptable to allow an HTTP/500 to happen? What do the web developers out there think?

Answer 1

There's really several questions here: should the web site display exceptions (No), should the web site display something more friendly to users (Yes), should the web site return a 500 error to Googlebot when it can't proceed (Probably), should you ask Googlebot to slow down (Yes), should you deliver 500 exception emails a day with no throttling or summarization (Probably not).

More detail:-

Using google.com/webmasters you can request that Google indexes your site less aggressively.

You should never show an Exception to users, you should always catch it and display a friendly error page BUT you need to be careful to preserve the HTTP code when you display that page (e.g. 404 or 500) since if you return a page with code = 200 then that error page will find its way into search engine indexes.

Any error handler should rate limit how often it sends email messages when errors occur.

A well written error handler should also allow the suppression of errors you know will happen - e.g. certain search engines insist on requesting pages that do not exist.

If Google can get you into a rate limit situation then high traffic from users may be able to do the same so overall it sounds like you need some kind of caching solution here too.

Answer 2

No, it's not acceptable. The web app should, at the very least, catch exceptions in global.asax.cs and do something reasonable with them.