Limiting asynchronous calls in Node.js

Question

I've got a Node.js app that gets a list of file locally and uploads them to a server. This list could contain thousands of files.

for (var i = 0; i < files.length; i++) {    upload_file(files[i]); } 

If I execute this with thousands of files, upload_file will get called thousands of times all at once, and most likely die (or at least struggle). In the synchronous world, we'd create a thread pool and limit it to a certain number of threads. Is there a simple way to limit how many asynchronous calls get executed at once?

Answer 1

As usual, I recommend Caolan McMahon's async module.

Make your upload_file function take a callback as it's second parameter:

var async = require("async");  function upload_file(file, callback) {     // Do funky stuff with file     callback(); }  var queue = async.queue(upload_file, 10); // Run ten simultaneous uploads  queue.drain = function() {     console.log("All files are uploaded"); };  // Queue your files for upload queue.push(files);  queue.concurrency = 20; // Increase to twenty simultaneous uploads