Rate limit GET requests

Question

I have an external API that rate-limits API requests to up to 25 requests per second. I want to insert parts of the results into a MongoDB database.

How can I rate limit the request function so that I don't miss any of API results for all of the array?

MongoClient.connect('mongodb://127.0.0.1:27017/test', function (err, db) {
    if (err) {
        throw err;
    } else {
        for (var i = 0; i < arr.length; i++) {
            //need to rate limit the following function, without missing any value in the arr array
            request( {
                method: 'GET',
                url: 'https://SOME_API/json?address='+arr[i]
            },
            function (error, response, body) {
                //doing computation, including inserting to mongo
            }
            )
        };
    }
});

Answer 1

This could possibly be done using the request-rate-limiter package. So you can add this to your code :

var RateLimiter = require('request-rate-limiter');
const REQS_PER_MIN = 25 * 60; // that's 25 per second
var limiter = new RateLimiter(REQS_PER_MIN);

and since request-rate-limiter is based on request you can just replace request with limiter.request

You can find further information on the package's npm page - https://www.npmjs.com/package/request-rate-limiter

On a personal note - I'd replace all these callbacks with promises

Answer 2

You need to combine 2 things.

• A throttling mechanism. I suggest _.throttle from the lodash project. This can do the rate limiting for you.
• You also need an async control flow mechanism to make sure the requests run in series (don't start second one until first one is done). For that I suggest async.eachSeries

Both of these changes will be cleaner if you refactor your code to this signature:

function scrape(address, callback) {
   //code to fetch a single address, do computation, and save to mongo here
   //invoke the callback with (error, result) when done
}