puppeteer: Access JSON response of a specific request as in the network tab of DevTools

Question

I'd like to directly get the response of the last HTTP request shown in the screenshot.

The current puppeteer code is shown below. Could anybody show me how to modify it so that it will get the JSON response directly from the browser? Thanks.

const puppeteer = require('puppeteer');

(async () => {
    //  const browser = await puppeteer.launch();
    const browser = await puppeteer.launch({
        headless: false
        , args: ['--user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3312.0 Safari/537.36"']
    });
    const page = await browser.newPage();
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');
    await page.goto('https://www.genecards.org/cgi-bin/carddisp.pl?gene=BSCL2');

    const linkHandlers = await page.$x("//div[@id='enhancers']//a[@data-track-event='Table See-All']");
    if (linkHandlers.length > 0) {
        await linkHandlers[0].click();
    } else {
        throw new Error("Link not found");
    }

    const html = await page.content()
    //await browser.close();
    console.log(html)
})();

Answer 1

You can use page.waitForResponse to wait for the response and response.json to parse the response as JSON.

Code

Replace the await linkHandlers[0].click(); part by this:

const [response] = await Promise.all([
    page.waitForResponse(response => response.url().includes('/gene/api/data/Enhancers')),
    linkHandlers[0].click()
]);
const dataObj = await response.json();
console.log(dataObj);

This will first wait for the response (while in parallel making the click). After the response is detected the response is parsed as JSON. To get the response result as plain text (instead of parsing it), you can use response.text()