What's the cleanest way to write a non-blocking for loop in javascript?

Question

So, I've been thinking about a brain teaser - what if I had a large object I for some reason had to iterate through in node js, and didn't want to block the event loop while I was doing that?

Here's an off-the-top-of-my-head example, I'm sure it can be much cleaner:

var forin = function(obj,callback){
    var keys = Object.keys(obj),
        index = 0,
        interval = setInterval(function(){
            if(index < keys.length){
                callback(keys[index],obj[keys[index]],obj);
            } else {
                clearInterval(interval);
            }
            index ++;
        },0);
}

While I'm sure there are other reasons for it being messy, this will execute slower than a regular for loop, because setInterval 0 doesn't actually execute every 0 ms, but I'm not sure how to make a loop with the much faster process.nextTick.

In my tests, I found this example takes 7 ms to run, as opposed to a native for loop (with hasOwnProperty() checks, logging the same info), which takes 4 ms.

So, what's the cleanest/fastest way to write this same code using node.js?

Answer 1

The behavior of process.nextTick has changed since the question was asked. The previous answers also did not follow the question as per the cleanliness and efficiency of the function.

// in node 0.9.0, process.nextTick fired before IO events, but setImmediate did
// not yet exist. before 0.9.0, process.nextTick between IO events, and after
// 0.9.0 it fired before IO events. if setImmediate and process.nextTick are
// both missing fall back to the tick shim.
var tick =
  (root.process && process.versions && process.versions.node === '0.9.0') ?
  tickShim :
  (root.setImmediate || (root.process && process.nextTick) || tickShim);

function tickShim(fn) {setTimeout(fn, 1);}

// executes the iter function for the first object key immediately, can be
// tweaked to instead defer immediately
function asyncForEach(object, iter) {
  var keys = Object.keys(object), offset = 0;

  (function next() {
    // invoke the iterator function
    iter.call(object, keys[offset], object[keys[offset]], object);

    if (++offset < keys.length) {
      tick(next);
    }
  })();
}

Do take note of @alessioalex's comments regarding Kue and proper job queueing.

See also: share-time, a module I wrote to do something similar to the intent of the original question.

Answer 2

There are many things to be said here.

• If you have a web application for example, you wouldn't want to do "heavy lifting" in that application's process. Even though your algorithm is efficient, it would still most probably slow down the app.
• Depending on what you're trying to achieve, you would probably use one of the following approaches:
  
  a) put your "for in" loop in a child process and get the result in your main app once it's over
  b) if you are trying to achieve something like delayed jobs (for ex sending emails) you should try https://github.com/LearnBoost/kue
  c) make a Kue-like program of your own using Redis to communicate between the main app and the "heavy lifting" app.
  

For these approaches you could also use multiple processes (for concurrency).

Now time for a sample code (it may not be perfect, so if you have a better suggestion please correct me):

var forIn, obj;

// the "for in" loop
forIn = function(obj, callback){
  var keys = Object.keys(obj);
  (function iterate(keys) {
    process.nextTick(function () {
      callback(keys[0], obj[keys[0]]);
      return ((keys = keys.slice(1)).length && iterate(keys));
    });
  })(keys);
};

// example usage of forIn
// console.log the key-val pair in the callback
function start_processing_the_big_object(my_object) {
  forIn(my_object, function (key, val) { console.log("key: %s; val: %s;", key, val); });
}

// Let's simulate a big object here
// and call the function above once the object is created
obj = {};
(function test(obj, i) {
  obj[i--] = "blah_blah_" + i;
  if (!i) { start_processing_the_big_object(obj); }
  return (i && process.nextTick(function() { test(obj, i); }));
})(obj, 30000);