Puppeteer exposeFunction which uses non-serializable argument

Question

I'm trying to use a 3rd party library which performs some complex DOM parsing:

/** 
 *  // Simplified for this example
 *  module.exports.parse = (document) => {return document.title; }
 */

const { parse } = require('./parse.js');

When I try and expose and evaluate the function in puppeteer:

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://stackoverflow.com', { waitUntil: ['domcontentloaded'] });
await page.exposeFunction('parse', (document) => {
    return parse(document);
});

await page.evaluate(() => {
    return window.parse(window.document);
});

I get an error:

Evaluation failed: TypeError: Converting circular structure to JSON\n at JSON.stringify ()\n at window.(anonymous function) (puppeteer_evaluation_script:13:22)\n at puppeteer_evaluation_script:3:31

In the documentation's example, it is passing a string (which can be serialized). Is there any known way to evaluate node.js methods which take window or document as its arguments?

Answer 1

This method explained below is a bit different so use with caution.

We can take the script to the browser instead of taking the dom to the node context. It is possible with power of webpack or browserify. Also this way, we won't need to serialize any circular variable.

Here is the minimal webpack config.

const path = require("path");

module.exports = {
  entry: "./browser/src.js",

  output: {
    path: path.resolve("browser"),
    filename: "dist.js",
    libraryTarget: "global"
  },

  module: {
    rules: [
      {
        test: /\.js$/,
        use: "babel-loader"
      }
    ]
  }
};

We will put the browser script on browser/src.js folder. Then when we run webpack, it will generate browser/dist.js which we can inject on the browser.

Finally I can call it with one of the following,

await page.addScriptTag({path: "./browser/dist.js"});
await page.evaluate(fs.readFileSync("./browser/dist.js", 'utf8'));

It works perfectly as long as we do not use any native binary file which is not possible to be bundled.

For simplicity, here are the other files,

// browser/src.js
module.exports.parse = window => {
  return window.location.href;
};

// index.js
const fs = require("fs");
const puppeteer = require("puppeteer");
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.evaluate(fs.readFileSync("./browser/dist.js", 'utf8'));
  const data = await page.evaluate(() => {
    return parse(window);
  });

  console.log({ data });
})();

// The result:
{ data: 'about:blank' }