Promise.all consumes all my RAM

Question

I've got a rate limiter for an API I am using which allows 20 requests per second. All requests are promise based and the promise will be resolved with the API data once there is a response.

The problem:

I setup a promiseArray which contains 58k promises all waiting for a response. So slowly the memory is increasing until I am running out of memory. In my specific situation I don't need to pass the resolved data to my then() and the data is eating up all my RAM.

The code:

  }).then(() => {
    // 2. Crawl for all clanprofiles from these leaderboards
    const promiseArray = []
    for (let i = 0; i < clanTags.length; i++) {
      // Resolved data from getClanProfile() is eating up all my RAM
      const p = backgroundScheduler.getClanProfile(clanTags[i], true)
      promiseArray.push(p)
    }
    return Promise.all(promiseArray)
  }).then(() => {

So is there a way to await until the promiseArray is resolved without needing the resolved data?

Answer 1

You will use a lesser amount of memory if you don't ever have 58k promises, their associated async operations and their result data active at once.

Instead you want to run X operations at once and then when one finishes, you start the next one with never more than X in flight at the same time and never more than X promises in use at once.

You can experiment with an appropriate value of X. A value of 1 is sequential operations but you can often improve overall end-to-end operation time by using some higher value of X. If all requests are hitting the same host, then X is probably no more than 5-10 (since a given host can't really do a lots of things at once and asking it to do more than it can do at once just slows it down).

If every request is to a different host, then you may be able to make X higher. Experimentation would give you an optimal value for both peak memory usage and overall throughput and somewhat depends upon your specific circumstances.

Bluebird's Promise.map() has a concurrency option that will do this for you, but there are also numerous ways to code for only X in flight at the same time.

Here are some other coding examples of managing how many are in flight at a time:

Make several requests to an API that can only handle 20 request a minute

How to execute promises in series?

unable to complete promises due to out of memory

Fire off 1,000,000 requests 100 at a time

How to make it so that I can execute say 10 promises at a time in javascript to prevent rate limits on api calls?

---

If you don't need the resolved data, you can allow it to be GCed sooner by replacing it like this:

  const p = backgroundScheduler.getClanProfile(clanTags[i], true).then(data => {
      return 0;     // make resolved value just be a simple number
                    // so other data is now eligible for GC
  });
  promiseArray.push(p)    

---

And, here's a simple implementation that iterates an array with no more than X requests in flight at the same time:

// takes an array of items and a function that returns a promise
// runs no more than maxConcurrent requests at once
function mapConcurrent(items, maxConcurrent, fn) {
    let index = 0;
    let inFlightCntr = 0;
    let doneCntr = 0;
    let results = new Array(items.length);
    let stop = false;
    
    return new Promise(function(resolve, reject) {
        
        function runNext() {
            let i = index;
            ++inFlightCntr;
            fn(items[index], index++).then(function(val) {
                ++doneCntr;
                --inFlightCntr;
                results[i] = val;
                run();
            }, function(err) {
                // set flag so we don't launch any more requests
                stop = true;
                reject(err);
            });
        }
        
        function run() {
            // launch as many as we're allowed to
            while (!stop && inFlightCntr < maxConcurrent && index < items.length) {
                runNext();
            }
            // if all are done, then resolve parent promise with results
            if (doneCntr === items.length) {
                resolve(results);
            }
        }
        
        run();
    });
}