net::ERR_TOO_MANY_REDIRECTS if I want to open website in puppeteer

Question

When I try to open a washington post article in puppeteer, I get error:

Error: net::ERR_TOO_MANY_REDIRECTS

internal/process/warning.js:27 (node:70458) UnhandledPromiseRejectionWarning: Error: net::ERR_TOO_MANY_REDIRECTS at https://www.washingtonpost.com/news/acts-of-faith/wp/2017/06/30/trump-promised-to-destroy-the-johnson-amendment-congress-is-targeting-it-now/
    at navigate (.../node_modules/puppeteer/lib/FrameManager.js:121:37)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
  -- ASYNC --
    at Frame.<anonymous> (.../node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (.../node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (.../node_modules/puppeteer/lib/helper.js:112:23)
    at .../app.js:255:14
    at processTicksAndRejections (internal/process/task_queues.js:89:5)

app.js

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://www.washingtonpost.com/news/acts-of-faith/wp/2017/06/30/trump-promised-to-destroy-the-johnson-amendment-congress-is-targeting-it-now/');

  const html = await page.content()
  console.log(html)
  await browser.close()
})()

I tried to abort navigation redirects based on this, but then I get Error: net::ERR_FAILED

internal/process/warning.js:27 (node:70488) UnhandledPromiseRejectionWarning: Error: net::ERR_FAILED at https://www.washingtonpost.com/news/acts-of-faith/wp/2017/06/30/trump-promised-to-destroy-the-johnson-amendment-congress-is-targeting-it-now/
    at navigate (.../node_modules/puppeteer/lib/FrameManager.js:121:37)
    at processTicksAndRejections (internal/process/task_queues.js:89:5)
  -- ASYNC --
    at Frame.<anonymous> (.../node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (.../node_modules/puppeteer/lib/Page.js:674:49)
    at Page.<anonymous> (.../node_modules/puppeteer/lib/helper.js:112:23)
    at .../app.js:247:14
    at processTicksAndRejections (internal/process/task_queues.js:89:5)

Answer 1

Ok, let's try the following:

1. Try seeing what is going on on the redirectChain.

2. Check the response.status() to see if it returns a redirect code (3xx).

3. Try faking some headers inside your request. Some bigger sites usually prevents APIs from digging around. Using User-Agent can help with this.