Can someone explain an ENOBUFS error?

Question

I'm making a bunch calls to a database that contains a large amount of data on a Windows 7 64 bit OS. As the calls are queuing up I get the error (for ever HTTP call after the first error):

Error: connect ENOBUFS *omitted* - Local (undefined:undefined)

From my google searching I've learned that this error means that my buffer has grown too large and my system's memory can no longer handle the buffer's size.

But I don't really understand what this means. I'm using node.js to with an HTTPS library to handle my requests. When the requests are getting queued and the sockets are opening is the buffer's size allocated in RAM? What will allow the buffer to expand to a greater size? Is this simply a hardware limitation?

I've also read that some OS are able to handle the size of the buffer better than other OS's. Is this the case? If so which OS would be better suited for running a node script that needs to fetch a lot of data via HTTPS requests?

Here's how I'm doing my requests.

for (let j = 0; j < dataQueries; j++) {
 getData(function())
}

function getData(callback){
 axios.get(url, config)
   .then((res) => {
      // parse res 
      callback(parsedRes(res))
   }).catch(function(err) {
      console.log("Spooky problem alert! : " + err);
   })
}

I've omitted some code for brevity, but this is generally how I'm doing my requests. I have a for loop that for every iteration launches a GET request via axios.

I know there is an axios.all command that is used for storing the promise the axios.HTTPMethod gives you, but I saw no change in my code when I set it up to store promises and then iterate over the promises via axios.all

Answer 1

In my case this got resolved by deleting the autogenerated zip files from my workspace, which got created every time I did cdk deploy. Turns out that my typescript compiler treated these files as source files and counted them into the tarball.

Answer 2

Thanks @Jonasw for your help, but there is a very simple solution to this problem. I used the small library throttled-queue to get the job done. (If you look at the source code it would be pretty easy to implement your own queue based on this package.

My code changed to:

const throttledQueue = require('throttled-queue')

let throttle = throttledQueue(15, 1000) // 15 times per second

for (let j = 0; j < dataQueries; j++) {\
 throttle(function(){
   getData(function(res){
     // do parsing
   })
 }

}

function getData(callback){
 axios.get(url, config)
   .then((res) => {
      // parse res 
      callback(parsedRes(res))
   }).catch(function(err) {
      console.log("Spooky problem alert! : " + err);
   })
}