Getting Axios Error: connect ETIMEDOUT when making high volume of calls

Question

I get this error when making a lot of calls from my Azure Function. What does this error mean? How to troubleshoot? My guess is I'm running out of TCP sockets? I don't really know how to check that in the Function App menu. However I checked the Logs for the Azure Maps API and there is no record of errors or dropped calls so I think it's definitely an axios/function issue.

I have seen some suggestions to add Connection:Keep-Alive header or to even create an axios instance and reuse it. However I am not sure if my problem is even related to that.

Error: connect ETIMEDOUT 13.107.42.21:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1141:16) {   errno: 'ETIMEDOUT',   code: 'ETIMEDOUT',   syscall: 'connect',   address: '13.107.42.21',   port: 443,   config: {
    url: 'https://atlas.microsoft.com/search/fuzzy/json?api-version=1.0&subscription-key= myKey &query=myAddress USA',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'x-ms-client-id': 'my client ID',
      'User-Agent': 'axios/0.19.2'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    validateStatus: [Function: validateStatus],
    host: 'atlas.microsoft.com',
    data: undefined   },   request: Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      bufferedRequest: null,
      lastBufferedRequest: null,
      pendingcb: 0,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: false,
      bufferedRequestCount: 0,
      corkedRequestsFree: [Object]
    },
    writable: true,
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    _options: {
      protocol: 'https:',
      maxRedirects: 21,
      maxBodyLength: 10485760,
      path: '/search/fuzzy/json?api-version=1.0&subscription-key= myKey &query=myAddress%20USA',
      method: 'GET',
      headers: [Object],
      agent: undefined,
      agents: [Object],
      auth: undefined,
      hostname: 'atlas.microsoft.com',
      port: null,
      nativeProtocols: [Object],
      pathname: '/search/fuzzy/json',
      search: '?api-version=1.0&subscription-key= myKey &query=myAddress%20USA'
    },
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function],
    _currentRequest: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [TLSSocket],
      connection: [TLSSocket],
      _header: 'GET /search/fuzzy/json?api-version=1.0&subscription-key= myKey &query=myAddress%20USA HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'x-ms-client-id: my client ID\r\n' +
        'User-Agent: axios/0.19.2\r\n' +
        'Host: atlas.microsoft.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      insecureHTTPParser: undefined,
      path: '/search/fuzzy/json?api-version=1.0&subscription-key= myKey &query=myAddress%20USA',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      _redirectable: [Circular],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    _currentUrl: 'https://atlas.microsoft.com/search/fuzzy/json?api-version=1.0&subscription-key= myKey &query=myAddress%20USA',
    [Symbol(kCapture)]: false   },   response: undefined,   isAxiosError: true,   toJSON: [Function] }

Answer 1

So essentially after talking to support for a whole month, they told me I was having SNAT problems. Basically running out of outbound ports... which didn't make sense to me since I was using a single axios instance and sharing it across the board. However after reading the provided documentation I came to the conclusion some additional changes needed to be made.

So I had a file in my main folder (in the wwwroot folder) that was called getAxios.js, this file would create a connection to a base url.

Directory layout:

-wwwroot

--getAxios.js

--myFunction

getAxios.js code:

const axios = require('axios')
const https = require('https')
const domain = 'https://atlas.microsoft.com'
let instance

module.exports = function (context)
{
    if (!instance)
    {
        //create axios instance
        instance = axios.create({
            baseURL: domain,
            timeout: 60000, //optional
            httpsAgent: new https.Agent({ keepAlive: true }),
            headers: {'Content-Type':'application/xml'}
        })
    }

    return instance;
}

However I didn't have the timeout set to anything specific before and I was not specifying keepAlive:true . Adding those two additional parameters has fixed my issue.

To explain a little more about the ports being exhausted, when a request is made and keepAlive is false that port connection is closed, but it has a small timeout duration before it can be used again. When you are processing a high volume of requests a bunch of these ports are in this state and can't be used and you end up having this issue. This is what MS support explained to me. I am sure I may be miswording a good bit of the networking aspect.

Answer 2

As per @search-learn, I solved my problem by explicitly specifying the timeout and httpsAgent properties on the Axios instance.

import axios from "axios";
const https = require("https");

const auth: string = Buffer.from(`:${envconfig.PAT}`).toString("base64");
axios.defaults.headers.common["Authorization"] = `Basic ${auth}`;
axios.defaults.timeout = 30000;
axios.defaults.httpsAgent = new https.Agent({ keepAlive: true });