Javascript based dynamic content using htmlUnit

Question

I have been stuck in getting JavaScript based dynamic content using HtmlUnit. I am expecting to get (Signin, Registration html content) from the page. With the following code, I only get the static content.

I am new to HtmlUnit. Any help will be highly appreciated.

String strURL = "https://www.checkmytrip.com" ;
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.waitForBackgroundJavaScript(60 * 1000);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));

String theContent = myPage.getWebResponse().getContentAsString();
System.out.println(theContent);      

Answer 1

Two points:

1. You need to waitForBackgroundJavaScript() after you get the page, as hinted here

2. You should use myPage.asText() or .asXml() instead, because getWebResponse() returns the original content without JavaScript execution.

   String strURL = "https://www.checkmytrip.com" ;
   java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
   java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);
   
   try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31)) {
       webClient.setAjaxController(new NicelyResynchronizingAjaxController());
   
       HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));
       webClient.waitForBackgroundJavaScript(10 * 1000);
   
       String theContent = myPage.asXml();
       System.out.println(theContent);
   }