Is there a faster way to check if an external web page exists?

Question

I wrote this method to check if a page exists or not:

protected bool PageExists(string url)
{
try
    {
        Uri u = new Uri(url);
        WebRequest w = WebRequest.Create(u);

            w.Method = WebRequestMethods.Http.Head;

        using (StreamReader s = new StreamReader(w.GetResponse().GetResponseStream()))
        {
            return (s.ReadToEnd().Length >= 0);
        }
    }
        catch
    {
        return false;
        }
    }

I am using it to check a set of pages (iterates from AAAA-AAAZ), and it takes between 3 and 7 seconds to run the entire loop. Is there a faster or more efficient way to do this?

Answer 1

I think your approach is rather good, but would change it into only downloading the headers by adding w.Method = WebRequestMethods.Http.Head; before calling GetResponse.

This could do it:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.example.com");
request.Method = WebRequestMethods.Http.Head;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
bool pageExists = response.StatusCode == HttpStatusCode.OK;

You may probably want to check for other status codes as well.