How to handle async Node.js in a loop

Question

I have such a loop :

var i,j,temparray,chunk = 200;
for (i=0,j=document.mainarray.length; i<j; i+=chunk) {
  temparray = document.mainarray.slice(i,i+chunk);

  var docs =  collection.find({ id: { "$in": temparray}}).toArray();

  docs.then(function(singleDoc)
  {
    if(singleDoc)
    {
      console.log("single doc length : " + singleDoc.length);
      var t;
      for(t = 0, len = singleDoc.length; t < len;t++)
      {
        fs.appendFile("C:/Users/x/Desktop/names.txt", singleDoc[t].name + "\n", function(err) {
          if(err) {
            return console.log(err);
          }
        });
      }
    }
  });
}

The loop iterates for two times. In first iteration it gets 200 elements, in second, it gets 130 elements. And when I open the .txt file, I see only 130 names. I guess because of the async nature of Node.js, only second part of the array is processed. What should I do to get all parts of the array to be processed? Thanks in advance.

EDIT : I finally turned the code to this :

var generalArr = [];
var i,j,temparray,chunk = 200;
for (i=0,j=document.mainarray.length; i<j; i+=chunk) {
    temparray = document.mainarray.slice(i,i+chunk);

generalArr.push(temparray);


} 

async.each(generalArr, function(item, callback)
{

  var docs =  collection.find({ id: { "$in": item}}).toArray();

   docs.then(function(singleDoc)
  {
    if(singleDoc)
    {
      console.log("single doc length : " + singleDoc.length);
              var t;
        for(t = 0, len = singleDoc.length; t < len;t++)
        {    
           fs.appendFile("C:/Users/x/Desktop/names.txt", singleDoc[t].name + "\n", function(err) {
          if(err) {
          return console.log(err);
          }
        });
        }
    }


  });

  callback(null);
})

When I change this line :

var docs =  collection.find({ id: { "$in": item}}).toArray();

To this line :

var docs =  collection.find({ id: { "$in": item}}).project({ name: 1 }).toArray();

It works, I'm able to print all names. I guess there is a problem with memory when I try without .project(). How can I make this work without using project? Should I change some memory limits? Thanks in advance.

Answer 1

I think your code is unnecessary complicated and appending file in a loop is very expensive when compared to in-memory computation. A better way would be to write to file just once.

var i, j, temparray, chunk = 200;
for (i = 0, j = document.mainarray.length; i < j; i += chunk) {
  temparray = document.mainarray.slice(i, i + chunk);
  generalArr.push(temparray);
}
const queryPromises = [];
generalArr.forEach((item, index) => {
  queryPromises.push(collection.find({ id: { "$in": item } }).toArray());
});
let stringToWrite = '';
Promise.all(queryPromises).then((result) => {
  result.forEach((item) => {
    item.forEach((element) => {
      //create a single string which you want to write
      stringToWrite = stringToWrite + "\n" + element.name;
    });
  });
  fs.appendFile("C:/Users/x/Desktop/names.txt", stringToWrite, function (err) {
    if (err) {
      return console.log(err);
    } else {
      // call your callback or return
    }
  });
});

In the code above, I do the following.

1. Wait for all the db queries to finish
2. Lets iterate over this list and create one string that we need to write to the file
3. Write to the file

Answer 2

Once you go asynchronous you cannot go back - all your code needs to be asynchronous. In node 8 you handle this with async and await keywords. In older versions you can use Promise - async/await are just syntax sugar for it anyway.

However, most of the API in node are older than Promise, and so they use callbacks instead. There is a promisify function to update callback functions to promises.

There are two ways to handle this, you can let all the asynchronous actions happen at the same time, or you can chain them one after another (which preserves order but takes longer).

So, collection.find is asynchronous, it either takes a callback function or returns a Promise. I'm going to assume that the API you're using does the latter, but your problem could be the former (in which case look up promisify).

var findPromise =  collection.find({ id: { "$in": item}});

Now, at this point findPromise holds the running find action. We say this is a promise that resolves (completes successfully) or rejects (throws an error). We want to queue up an action to do once it completes, and we do that with then:

// The result of collection.find is the collection of matches
findPromise.then(function(docs) {
    // Any code we run here happens asynchronously
});

// Code here will run first

Inside the promise we can return further promises (allowing them to be chained - complete one async, then complete the next, then fire the final resolve once all done) or use Promise.all to let them all happen in parallel and resolve once done:

var p = new Promise(function(resolve, reject) {
    var findPromise =  collection.find({ id: { "$in": item}});
    findPromise.then(function(docs) {
        var singleDocNames = [];
        for(var i = 0; i < docs.length; i++) {
            var singleDoc = docs[i];
            if(!singleDoc)
                 continue;

            for(var t = 0; t < singleDoc.length; t++)
                singleDocNames.push(singleDoc[t].name);
        }

        // Resolve the outer promise with the final result
        resolve(singleDocNames);
    });
});

// When the promise finishes log it to the console
p.then(console.log);

// Code inline here will fire before the promise

This is much easier in node 8 with async/await:

async function p() {
    // Await puts the rest of this function in the .then() of the promise
    const docs = await collection.find({ id: { "$in": item}});

    const singleDocNames = [];
    for(var i = 0; i < docs.length; i++) {
        // ... synchronous code unchanged ...
    }

    // Resolve the outer promise with the final result
    return singleDocNames;
});

// async functions can be treated like promises
p().then(console.log);

If you need to write the results to a text file asynchronously there are a couple of ways to do it - you can wait until the end and write all of them, or chain a promise to write them after each find, though I find parallel IO operations tend to be at more risk of deadlocks.