Using rejectUnauthorized with node-fetch in node.js

Question

I currently use request to make http requests in node.js. I had at some point encountered an issue where I was getting errors that indicated UNABLE_TO_GET_ISSUER_CERT_LOCALLY. To get around that it set rejectUnauthorized. My working code with request looks like this:

    var url = 'someurl';
    var options = {
        url: url,
        port: 443,
        // proxy: process.env.HTTPS_PROXY, -- no need to do this as request honors env vars
        headers: {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
            'Accept-Language': 'en-us',
            'Content-Language': 'en-us'
        },
        timeout: 0,
        encoding: null,
        rejectUnauthorized: false // added this to prevent the UNABLE_TO_GET_ISSUER_CERT_LOCALLY error
    };
    request(options, function (err, resp, body) {
        if (err) reject(err);
        else resolve(body.toString());
    });

I thought I would try switching to the fetch api using async/await and am now trying to use node-fetch to do the same thing. However, when I do the same thing I am back to the UNABLE_TO_GET_ISSUER_CERT_LOCALLY errors. I read that I needed to use a proxy agent and tried using the proxy-agent module but I am still not having any luck.

Based off of the post https://github.com/TooTallNate/node-https-proxy-agent/issues/11 I thought the following would work:

    var options = {
        headers: {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko',
            'Accept-Language': 'en-us',
            'Content-Language': 'en-us'
        },
        timeout: 0,
        encoding: null
    };
    var proxyOptions = nodeurl.parse(process.env.HTTPS_PROXY);
    proxyOptions.rejectUnauthorized = false;
    options.agent = new ProxyAgent(proxyOptions);
    const resp = await fetch('someurl', options);
    return await resp.text();

but I still get the same error. So far the only way I've been able to get around this using node-fetch is to set NODE_TLS_REJECT_UNAUTHORIZED=0 in my environment which I don't really want to do. Can someone help show me how to get rejectUnauthorized to work with node-fetch (presumably using an agent but I don't honestly care how as long as it's specified as part of the request).

Answer 1

This is how I got this to work using rejectUnauthorized and the Fetch API in a Node.js app.

Keep in mind that using rejectUnauthorized is dangerous as it opens you up to potential security risks, as it circumvents a problematic certificate.

const fetch = require("node-fetch");
const https = require('https');

const httpsAgent = new https.Agent({
  rejectUnauthorized: false,
});

async function getData() {
  const resp = await fetch(
    "https://myexampleapi.com/endpoint",
    {
      agent: httpsAgent,
    },
  )
  const data = await resp.json()
  return data
}

Answer 2

Use proxy

You should know that node-https-proxy-agent latest version have a problem and doesn't work with Fetch! You can use older version 3.x and down! And it will work! Otherwise Better you can use the node-tunnel module https://www.npmjs.com/package/tunnel! You can too use the wrapping module proxy-http-agent that is based on node-tunnel https://www.npmjs.com/package/proxy-http-agent! That provide automatic detection of protocol for the proxy! One method for all! And more options and affinity! And both of them support both http and https !

You can see the usage and see a good example of proxy building and setup in this module and repo (check the tests): https://www.npmjs.com/package/net-proxy https://github.com/Glitnirian/node-net-proxy#readme

ex:

import { ProxyServer } from 'net-proxy';
import { getProxyHttpAgent } from 'proxy-http-agent';

// ...

// __________ setting the proxy

const proxy = new ProxyServer({
    port: proxyPort
});
 
proxy.server.on('data', (data: any) => { // accessing the server instance
    console.log(data);
});
 
await proxy.awaitStartedListening(); // await server to start
 
// After server started
 
// ______________ making the call through the proxy to a server through http:

let proxyUrl = `http://localhost:${proxyPort}`; // Protocol from the proxy is automatically detected

let agent = getProxyHttpAgent({
    proxy: proxyUrl,
    endServerProtocol: 'http:' // the end server protocol (http://localhost:${localApiServerPort} for example)
});
 
const response = await fetch(`http://localhost:${localApiServerPort}`, {
    method: 'GET',
    agent 
});

// ___________________ making a call through the proxy to a server through https:

agent = getProxyHttpAgent({
    proxy: proxyUrl, // proxy as url string! We can use an object (as tunnel module require too)
    rejectUnauthorized: false // <==== here it go
});

const response2 = await fetch(`https://localhost:${localApiHttpsServerPort}`, {
    method: 'GET',
    agent
});

You can see more examples and details in the doc here:

https://www.npmjs.com/package/proxy-http-agent

And you can too use directly node-tunnel! But the package is just a simple wrapper! That make it more simpler!

Add rejectUnauthorized

For the one that doesn't know well!

As per this thread

https://github.com/node-fetch/node-fetch/issues/15

We use the https.Agent to pass the rejectUnauthorized parameter!

const agent = new https.Agent({
  key: fs.readFileSync(`${CERT_PATH}.key`),
  cert: fs.readFileSync(`${CERT_PATH}.crt`),
  rejectUnauthorized: false
})

A complete example

import https from "https";
const agent = new https.Agent({
  rejectUnauthorized: false
});
fetch(myUrl, { agent });

For fetch you can too use an environment variable as follow

process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

This way it gonna be set globaly and not per each call! Which may be more appropriate if you are using a constant proxy! For all calls! As when sitting behind the company proxy!

why

By default node fetch! And most of the http requests clients! All use the security and insure a valid ssl Certificate when using https! To disable this behavior we need to disable that check somehow! Depending on the libs it may be different!

For fetch that's how it's done!

With http.request! (underlying)

const https = require('https');

const options = {
  hostname: 'encrypted.google.com',
  port: 443,
  path: '/',
  method: 'GET',
  rejectUnauthorized: false /// <<<== here
};

const req = https.request(options, (res) => {
  console.log('statusCode:', res.statusCode);
  console.log('headers:', res.headers);

  res.on('data', (d) => {
    process.stdout.write(d);
  });
});

req.on('error', (e) => {
  console.error(e);
});
req.end();

check this:

https://nodejs.org/api/https.html#https_https_request_url_options_callback

Also it's part of tls.connect Options

Which you can check here

https://nodejs.org/api/tls.html#tls_tls_connect_options_callback