ignore-certificate-errors + headless puppeteer+google cloud

Question

The website I am trying to access has ssl certificate-errors

I am using this version of puppeteer "puppeteer": "1.13.0".

When I try to await page.goto('http://bad_ssl_certificate_website') I have timeout error on google cloud only.

TimeoutError: Navigation Timeout Exceeded:

However, It works perfectly fine locally on MAC.

I think the problem is ssl-certificate-errors for my website, because if I try with "google.com" it works okay in both environments. I used https://www.sslshopper.com to check ssl certificates,and It mentioned this.

The certificate is not trusted in all web browsers. You may need to install an Intermediate/chain certificate to link it to a trusted root certificate. Learn more about this error. You can fix this by following DigiCert's Certificate Installation Instructions for your server platform. Pay attention to the parts about Intermediate certificates.

When I was using older version of puppeteer I had problems locally as well. I saw the exactly the same error

'TimeoutError: Navigation Timeout Exceeded:'

Updating to the newest version of puppeteer has fixed only running the puppeteer locally, but it has not fixed the puppeteer running on google cloud

This is how I setup puppeteer to lunch.

   const browser = await puppeteer.launch({
      headless: true,
      ignoreHTTPSErrors: true,
      args: [
        "--proxy-server='direct://'",
        '--proxy-bypass-list=*',
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-setuid-sandbox',
        '--no-first-run',
        '--no-sandbox',
        '--no-zygote',
        '--single-process',
        '--ignore-certificate-errors',
        '--ignore-certificate-errors-spki-list',
        '--enable-features=NetworkService'
      ]
    });

I found some related issues: https://bugs.chromium.org/p/chromium/issues/detail?id=877075

Answer 1

Just turn .IgnoreHTTPSErrors = True in the LaunchAsync constructor.

Example:

await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
var browser = await Puppeteer.LaunchAsync(new LaunchOptions() {
    Headless: true,
    IgnoreHTTPSErrors: true
});

Answer 2

The --ignore-certificate-errors-spki-list actually accepts a whitelist of public key hashes ignore certificate-related errors. So it is used like: --ignore-certificate-errors-spki-list=jc7r1tE54FOO=

Chromium doc