Can't scrape contributors div from github insights page

Question

I am trying to scrape github contributor insights using NodeJS, puppeteer, and cheerio.

const cheerio = require('cheerio');
const puppeteer = require('puppeteer');
const url = 'https://github.com/grey-software/grey.software/graphs/contributors';

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.waitForSelector('#contributors', {
    visible: true,
  })

  await page.goto(url);
  const pageContent = await page.content()

  const $ = cheerio.load(pageContent);
  $('.contrib-person').each(function (i, elem) {
    console.log(elem)
  });
  await browser.close();
})();

I get the following error when I run the code above

UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector "#contributors" failed: timeout 30000ms exceeded

The #contributors div should load within the 30 seconds, but I always get this timeout. Note: page.waitForNavigation() gives the same timeout error

Answer 1

Your issue is you are waiting for the selector to exist... Before you go to the website in the first place. use page.waitForSelector() after you use page.goto()