Why does it show question marks on the message box instead of text

Question

Why does it shows question marks on the message box instead of text

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://teamxor.net/vb/tx48/"+ page);

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

StreamReader sr = new StreamReader(response.GetResponseStream());

string result = sr.ReadToEnd();

Regex r = new Regex("<div>.*?</div>");
MatchCollection mr = r.Matches(result);

foreach (Match m in mr)
{
    MessageBox.Show(m.Value, "Test", MessageBoxButtons.OK, MessageBoxIcon.Information, MessageBoxDefaultButton.Button1, MessageBoxOptions.RtlReading);
}      

Answer 1

The problem lies in the use of a non-default code page. Your HTML shows you are using code page 1256. You have to tell .NET that, else it thinks it is UTF-8:

StreamReader sr = new StreamReader( response.GetResponseStream()
                                  , Encoding.GetEncoding(1256) // <-- this one
                                  );

Use Encoding.GetEncoding to get the right code page. I suggest to use UTF8 instead, since that is easily recognized by .NET.

Answer 2

Web servers can return a response in whatever encoding they want, although they typically choose an encoding that matches the browser's preferred language.

The encoding used is returned as the charset element of the Content-Type header. In .NET you can retrieve the encoding used from the HttpWebResponse.CharacterSet property. You can use the returned charset to construct an Encoding object to use for reading the response:

var charset= response.CharacterSet;
var encoding = Encoding.GetEncoding(charset);
var sr= new StreamReader(response.GetResponseStream(),encoding);
....