Puppeteer Crawler - Error: net::ERR_TUNNEL_CONNECTION_FAILED

Question

Currently I have my Puppeteer running with a Proxy on Heroku. Locally the proxy relay works totally fine. I however get the error Error: net::ERR_TUNNEL_CONNECTION_FAILED. I've set all .env info in the Heroku config vars so they are all available.

Any idea how I can fix this error and resolve the issue?

I currently have

 const browser = await puppeteer.launch({
      args: [
      "--proxy-server=https=myproxy:myproxyport",
      "--no-sandbox",
      '--disable-gpu',
      "--disable-setuid-sandbox",
      ],
      timeout: 0,
      headless: true,
    });

Answer 1

page.authentication

The correct format for proxy-server argument is,

--proxy-server=HOSTNAME:PORT

If it's HTTPS proxy, you can pass the username and password using page.authenticate before even doing a navigation,

page.authenticate({username:'user', password:'password'});

Complete code would look like this,

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
        headless:false,
        ignoreHTTPSErrors:true,
        args: ['--no-sandbox','--proxy-server=HOSTNAME:PORT']
  });
  const page = await browser.newPage();

  // Authenticate Here 
  await page.authenticate({username:user, password:password});
  await page.goto('https://www.example.com/');
})();

Proxy Chain

If somehow the authentication does not work using above method, you might want to handle the authentication somewhere else.

There are multiple packages to do that, one is proxy-chain, with this, you can take one proxy, and use it as new proxy server.

The proxyChain.anonymizeProxy(proxyUrl) will take one proxy with username and password, create one new proxy which you can use on your script.

const puppeteer = require('puppeteer');
const proxyChain = require('proxy-chain');

(async() => {
    const oldProxyUrl = 'http://username:password@hostname:8000';
    const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);

    // Prints something like "http://127.0.0.1:12345"
    console.log(newProxyUrl);

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${newProxyUrl}`],
    });

    // Do your magic here...
    const page = await browser.newPage();
    await page.goto('https://www.example.com');
})();