Java: File type of `url.openStream()`

Question

I wrote this this method to download a webpage given a URL. It is designed to download HTML only. If I want to do error checking and allow HTML only how should I do this?

public static String download(URL url) throws IOException {
    InputStream is = url.openStream();
    BufferedReader reader = new BufferedReader(new InputStreamReader(is));
    String page = "";
    String line;    
    while((line = reader.readLine()) != null){
        page = page + line;
    }
    return page;
}

Originally I was planning on doing this:

String file = url.getFile();
if(file.subString(file.indexOf("."),file.length()-1).equalsIgnoreCase("HTML")){
    // do method

However the URL: http://www.smu.com returns "" for url.getFile(). Anyone have any suggestions?

Answer 1

To test if you're getting html you can use URL.openConnection() to get a UrlConnection can then call getContentType() which should return "text/html" for an HTML page. You can then use the getInputStream() method on the UrlConnection() as a drop in replacement for url.openStream();

If you actually want to validate that the content the server is sending you is HTML you'd need to find an HTML validation library. I don't know of one off-hand, sorry.

Something to consider, which may be why www.smu.com returns no data, is that a number of websites will serve different data depending on the User-Agent string sent on the HTTP connection. You may need to modify that on your UrlConnection with: UrlConnection.addRequestProperty("User-Agent", ...); See more info here : Setting user agent of a java URLConnection

Answer 2

If you want to check the content beyond checking the Content-Type header, then you can use an HTML parser such as (the misleadingly named!) JTidy.