Webview only showing raw HTML text on some pages

Question

I am working on a project where we are interested in intercepting some HTTP traffic from a WebView, then attach some extra headers to the requests and then get a return back and display it in the WebView.

For this, we use the shouldInterceptRequest(WebView view, String URL) method, which returns the WebResourceResponse that the webview is supposed to display.

Here is what I do:

private WebResourceResponse handleRequest(String url){
    Log.d(TAG, "Intercepting request at : " + url);
    HttpURLConnection connection = null;        
    URL serverAddress = null;           
    WebResourceResponse response = null;
    InputStream is = null;
    String type = null;
    String encoding = null;
    int statusCode;
    int length;

    try
    {
        //Set up the initial connection
        serverAddress = new URL(url);
        connection = (HttpURLConnection)serverAddress.openConnection();

        // Initiate a HEAD request to get meta data
        connection.setRequestMethod("HEAD");

        // Let the WebView handle requests with no headers in them.
        Map<String, List<String>> headers = connection.getHeaderFields();
        if(headers == null){
            Log.d(TAG, "Retrieved response with zero headers at: " + url);
            return null;
        }

        statusCode = connection.getResponseCode();

        if(statusCode == 200){
            // OK, delegate request if content length is large enough
            type = connection.getContentType();
            encoding = connection.getContentEncoding();
            length = connection.getContentLength();

            Log.d(TAG, "ContentType = " + type + " encoding = " + encoding + " " + url);

           connection.disconnect();

            if(length > SIZE_THRESHOLD){
                // send to other device

                Log.d(TAG, "Delegating request to other device NOT IMPLEMENTED");
                // TODO: Currently just handling them locally.
            }
            else{
                connection = (HttpURLConnection)serverAddress.openConnection();
                connection.setRequestMethod("GET");             
                is = connection.getInputStream();
                statusCode = connection.getResponseCode();
                type = connection.getContentType();
                encoding = connection.getContentEncoding();

                Log.d(TAG, "Done Loading " + url);

                return new WebResourceResponse(type, encoding, is);
            }

        }
    } 
    catch (MalformedURLException e) {
            e.printStackTrace();
    }
    catch (ProtocolException e) {
            e.printStackTrace();
    } 
    catch (IOException e) {
        Log.d(TAG, "IOException when connection to : " + url + " Stacktrace: " + Log.getStackTraceString(e));
    }
    return response;
}   

If I try to load a page like "www.madopskrifter.nu" or "www.newz.dk" then I get the whole content just fine with pictures displayed and everything. However, if I browse to pages like stackoverflow I just get the HTML index file at stackoverflow.com. It is the same for youtube.com. (I do have JavaScript enabled in the WebView settings)

Is there something I am missing? at the moment I am not even adding my own headers or anything, I am just plainly returning the same response, except that I am getting them "manually". (Also I am not handling requests with zero headers right now, but even if I comment out that code, then it the problem still exists).

Any help would be greatly appreciated,

Answer 1

I found a solution for this same problem I was having, YMMV:

My HttpUrlConnection was returning:

• encoding: null
• mimeType: "text/html; charset=utf-8"

That was the problem.

I now check for html files, and change to:

• encoding: "UTF-8"
• mimeType: "text/html"

Example:

String responseEncoding = urlConnection.getContentEncoding();
String responseMimeType = urlConnection.getContentType();

String html = "";
if (resourceUrl.endsWith("html"))
{
    responseEncoding = "UTF-8";
    responseMimeType = "text/html";
}

if((responseCode == 200) && (!is.equals("")))
{
    return new WebResourceResponse(responseMimeType, responseEncoding, returnStream); 
} else
{
    return null;
}