Puppeteer loads blank page with 429 when accessing URL

Question

I am working on a project to obtain pricing information from a hotel website, but I cannot perform any searches when loading the website in puppeteer.

Here is a snippet of my JavaScript that opens Chrome.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false, devTools: false });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3683.103 Safari/537.36');

  await page.goto('https://www.hyatt.com/');

})();

How are they detecting that I'm using Chrome controlled by Puppeteer, even though it is running a headful browser?

Thanks

Answer 1

They're fingerprinting you. It's some javascript that enumerates things like your browser plugins and navigator object and sends that back to the server.

So you'd have to intercept that post and change it to something a normal browser would send.

Answer 2

Try adding the following header which is expected

'accept-language' : 'en-US,en;q=0.9'

works for me in other languages.

From this answer it looks like you need something like:

await page.setExtraHTTPHeaders({
    'Accept-Language': 'en-US,en;q=0.9'
});

Answer 3

The page is using Fingerprintjs2 to analyse your browser before letting you access the page.

On your first request, the page is sending you to a special detection page, which is using Fingerprint2. Quote from the github repository:

Modern & flexible browser fingerprinting library

The code will check all kinds of browser information, like specific attributes, timezone, pixelRatio of your device, etc. You can find the actual source code in this file on github.

You will have to change all kind of browser attributes to make the fingerprinting library actually think you are a "normal user".