waitForSelector with Puppeteer not responding only in headless mode

Question

I am having a strange issue with puppeteer, I have the following code:

try {
    await Promise.all([
        page.waitForSelector(HomeSelector.profileButton),
        cursor.click(HomeSelector.profileButton),
        page.waitForNavigation({ waitUntil: 'networkidle2' })
    ]);
    // Never responds in headless mode
    await page.waitForSelector(ProfileSelector.nameField);


    resolve();
} catch(error) {
    reject(error); 
}

When I run puppeteer with headless: false it successfully waits for the selector nameField however with headless: true I just get the timeout error TimeoutError: waiting for selector .

I have tried changing the waitForNavigation parameter and that doesn't make a difference, I have also taken a screenshot of the website to confirm the elements are there.

Has anyone else faced this issue before? Any ideas how to go around fixing it?

Answer 1

The solution for me was to add the userAgent when creating the new context, I did this because when checking if the content of the page was being accessed with a screenshot await page.screenshot({ path: "screenshot.png" }); immediately after performing the await page.goto(url);

await page.goto(url);
await page.screenshot({ path: "screenshot.png" });
await page.waitForSelector("#user-rut");

This helps me realize that I was getting a 403 forbidden nginx error. Apparently running headless: true the puppeteer deals with it with another type of agent.

The solution for me was to specify the userAgent to be able to load the page correctly and avoid the error that was getting time out Error: page.waitForSelector waiting for selector to be visible

const context = await browser.newContext({
    userAgent:
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36",
  });