Open a connection with Jsoup, get status code and parse document

Question

I'm creating a class using jsoup that will do the following:

1. The constructor opens a connection to a url.
2. I have a method that will check the status of the page. i.e. 200, 404 etc.
3. I have a method to parse the page and return a list of urls.#

Below is a rough working of what I am trying to do, not its very rough as I've been trying a lot of different things

public class ParsePage {
private String path;
Connection.Response response = null;

private ParsePage(String langLocale){
    try {
        response = Jsoup.connect(path)
                .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
                .timeout(10000)
                .execute();
    } catch (IOException e) {
        System.out.println("io - "+e);
    }
}

public int getSitemapStatus(){
    int statusCode = response.statusCode();
    return statusCode;
}

public ArrayList<String> getUrls(){
    ArrayList<String> urls = new ArrayList<String>();

 }
}

As you can see I can get the page status, but using the already open connection from the constructor I don't know how to get the document to parse, I tried using:

Document doc = connection.get();

But that's a no go. Any suggestions? Or better ways to go about this?

Answer 1

As stated in the JSoup Documentation for the Connection.Response type, there is a parse() method that parse the response's body as a Document and returns it. When you have that, you can do whatever you want with it.

For example, see the implementation of getUrls()

public class ParsePage {
   private String path;
   Connection.Response response = null;

   private ParsePage(String langLocale){
      try {
         response = Jsoup.connect(path)
            .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
            .timeout(10000)
            .execute();
      } catch (IOException e) {
         System.out.println("io - "+e);
      }
   }

   public int getSitemapStatus() {
      int statusCode = response.statusCode();
      return statusCode;
   }

   public ArrayList<String> getUrls() {
      ArrayList<String> urls = new ArrayList<String>();
      Document doc = response.parse();
      // do whatever you want, for example retrieving the <url> from the sitemap
      for (Element url : doc.select("url")) {
         urls.add(url.select("loc").text());
      }
      return urls;
   }
}