WebRequest Strange NotFound Error

Question

I have 2 different ASP.NET Core websites: admin and public. Both running on staging server and on local machine.

I send GET request to different pages to determine execution time of different pages and encountered problem with admin site: all urls on local instance and staging always returns 404 error:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The remote server returned an error: (404) Not Found.

Meanwhile, same requests in browser return html pages normally. Requests through HttpWebRequest to public site always also return 200 Status Code (OK). Code for request I took here.

I tried to add all headers and cookies from browser request, but it didn't help. Also tried to debug local instance and found that no exceptions thrown while request executed.

Any ideas?

Answer 1

404 is way to generic. The code provided in answer in your link (https://stackoverflow.com/a/16642279/571203) does no error handling - this is brilliant example of how you can get to troubles when you blindly copy code from stackoverflow :)

Modified code with error handling should look like:

string urlAddress = "http://google.com/rrr";

var request = (HttpWebRequest)WebRequest.Create(urlAddress);
string data = null;
string errorData = null;
try
{
  using (var response = (HttpWebResponse)request.GetResponse())
  {
    data = ReadResponse(response);
  }
}
catch (WebException exception)
{
  using (var response = (HttpWebResponse)exception.Response)
  {
    errorData = ReadResponse(response);
  }
}

static string ReadResponse(HttpWebResponse response)
{
  if (response.CharacterSet == null)
  {
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
      return reader.ReadToEnd();
    }
  }
  using (var reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(response.CharacterSet)))
  {
    return reader.ReadToEnd();
  }
}

So when there is an exception, you'll get not just the status code, but entire response from server in the errorData variable.

One thing to check is proxy - browser can use http proxy while your server client uses none.