What is google's character encoding?

Question

Apparently Google's encoding is UTF-8 as it's stated in it's html meta tag. But when I open a search page for scharfes+s with ASP WebRequest.GetResponse(), it's full of unrecognized characters. Does someone know what's going on there?

For your convenience, code is pasted below

Asp Page

<form id="form1" runat="server">
<div>
    <div runat="server" id="output"/>
</div>
</form>

Codebehind

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net;
using System.IO;
using System.Text;

public partial class SearchEngineCaller : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        HttpWebRequest queryPage = (HttpWebRequest)WebRequest.Create("https://www.google.com/search?q=scharfes+s");
        queryPage.Credentials = CredentialCache.DefaultCredentials;

        HttpWebResponse response = (HttpWebResponse)queryPage.GetResponse();

        Stream receiveStream = response.GetResponseStream();
        StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
        output.InnerHtml = readStream.ReadToEnd();
    }
}

What encoding should I use?

Answer 1

You have to set some HTTP headers for the HttpWebRequest object:

HttpWebRequest queryPage = (HttpWebRequest)WebRequest.Create("https://www.google.com/search?q=scharfes+s");
queryPage.Credentials = CredentialCache.DefaultCredentials;
queryPage.Accept = "text/html";
queryPage.Headers["Accept-Charset"] = "utf-8";
queryPage.UserAgent = "Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/21.0";

IMPORTANT: Setting the Accept-Charset is not enough, it's important to set the User-Agent, too (I copied the above user agent string from here). I tried this solution, and it works for me (test code).