Retrieving JavaScript Rendered HTML with Puppeteer

Question

I am attempting to scrape the html from this NCBI.gov page. I need to include the #see-all URL fragment so that I am guaranteed to get the searchpage instead of retrieving the HTML from an incorrect gene page https://www.ncbi.nlm.nih.gov/gene/119016.

URL fragments are not passed to the server, and are instead used by the javascript of the page client-side to (in this case) create entirely different HTML, which is what you get when you go to the page in a browser and "View page source", which is the HTML I want to retrieve. R readLines() ignores url tags followed by #

I tried using phantomJS first, but it just returned the error described here ReferenceError: Can't find variable: Map, and it seems to result from phantomJS not supporting some feature that NCBI was using, thus eliminating this route to solution.

I had more success with Puppeteer using the following Javascript evaluated with node.js:

const puppeteer = require('puppeteer');
(async() => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(
    'https://www.ncbi.nlm.nih.gov/gene/?term=AGAP8#see-all');
  var HTML = await page.content()
  const fs = require('fs');
  var ws = fs.createWriteStream(
    'TempInterfaceWithChrome.js'
  );
  ws.write(HTML);
  ws.end();
  var ws2 = fs.createWriteStream(
    'finishedFlag'
  );
  ws2.end();
  browser.close();
})();

however this returned what appeared to be the pre-rendered html. how do I (programmatically) get the final html that I get in browser?

Answer 1

You can try to change this:

await page.goto(
  'https://www.ncbi.nlm.nih.gov/gene/?term=AGAP8#see-all');

into this:

  await page.goto(
    'https://www.ncbi.nlm.nih.gov/gene/?term=AGAP8#see-all', {waitUntil: 'networkidle'});

Or, you can create a function listenFor() to listen to a custom event on page load:

function listenFor(type) {
  return page.evaluateOnNewDocument(type => {
    document.addEventListener(type, e => {
      window.onCustomEvent({type, detail: e.detail});
    });
  }, type);
}`

await listenFor('custom-event-ready'); // Listen for "custom-event-ready" custom event on page load.

LE:

This also might come in handy:

await page.waitForSelector('h3'); // replace h3 with your selector

Answer 2

Maybe try to wait

await page.waitForNavigation(5);

and after

let html = await page.content();

Answer 3

I had success using the following to get html content that was generated after the page has been loaded.

const browser = await puppeteer.launch();
try {
  const page = await browser.newPage();
  await page.goto(url);
  await page.waitFor(2000);
  let html_content = await page.evaluate(el => el.innerHTML, await page.$('.element-class-name'));
  console.log(html_content);
} catch (err) {
  console.log(err);
}

Hope this helps.