HttpWebRequest error 403

Question

I am new in C# and have requirement to retrieve a url from C#. Most of case it’s working fine but in one case its throws an error. url is as follows http://whois.afrinic.net/cgi-bin/whois?searchtext=41.132.178.138

Error is:

Exception in the HTTP request for url: http://whois.afrinic.net/cgi-bin/whois?searchtext=41.132.178.138 The remote server returned an error: (403) Forbidden.

My code is as

void MyFUnction(string url)
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

    request.UserAgent = ".NET Framework Test Client";
    request.ContentType = "application/x-www-form-urlencoded";
    Logger.WriteMyLog("application/x-www-form-urlencoded");


    // execute the request
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    // we will read data via the response stream
    Stream resStream = response.GetResponseStream();

    string tempString = null;
    int count = 0;
    do
    {
        // fill the buffer with data
        count = resStream.Read(buf, 0, buf.Length);

        // make sure we read some data
        if (count != 0)
        {
            // translate from bytes to ASCII text
            tempString = Encoding.ASCII.GetString(buf, 0, count);
            if (httpData == null)
                httpData = tempString;
            else
                httpData += tempString;

        }
    }
    while (count > 0); // any more data to read?
}

Answer 1

Remove your ContentType line.

request.ContentType....

You're not doing a form post, only retrieving the page with "GET".

request.Method = "GET"; //this is the default behavior

And also set the Accept property to "text/html".

request.Accept = "text/html";

Answer 2

Set

request.Accept = "text/html";

as well and it will work.

I don't know why they are configured that way, might be deliberate to discourage some bots. You might want to check their terms of service whether you may query their site automatically.

---

Edited to add: Still convinced that my answer was correct in this case, I can reproduce the 403 every time if I don't set the Accept header. The ContentType is superfluous, but not harmful.
In any case, you'll also want to change your function to properly dispose the response, and to read the response with the correct character encoding:

void MyFunction(string url)
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
    request.UserAgent = ".NET Framework Test Client";
    request.Accept = "text/html";
    Logger.WriteMyLog("application/x-www-form-urlencoded");

    // execute the request
    using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
    {
        // we will read data via the response stream
        Stream resStream = response.GetResponseStream();
        StreamReader streamReader = new StreamReader(
                                      resStream,
                                      Encoding.GetEncoding(response.CharacterSet)
                                    );
        httpData = streamReader.ReadToEnd();
    }
}