What are ES6 generators and how can I use them in node.js?

Question

I was at a node.js meetup today, and someone I met there said that node.js has es6 generators. He said that this is a huge improvement over callback style programming, and would change the node landscape. Iirc, he said something about call stack and exceptions.

I looked them up, but haven't really found any resource that explains them in a beginner-friendly way. What's a high-level overview of generators, and how are the different (or better?) than callbacks?

PS: It'd be really helpful if you could give a snippet of code to highlight the difference in common scenarios (making an http request or a db call).

Answer 1

Generators, fibers and coroutines

"Generators" (besides being "generators") are also the basic buildings blocks of "fibers" or "coroutines". With fibers, you can "pause" a function waiting for an async call to return, effectively avoiding to declare a callback function "on the spot" and creating a "closure". Say goodbye to callback hell.

Closures and try-catch

...he said something about call stack and exceptions

The problem with "closures" is that even if they "magically" keep the state of the local variables for the callback, a "closure" can not keep the call stack.

At the moment of callback, normally, the calling function has returned a long time ago, so any "catch" block on the calling function cannot catch exceptions in the async function itself or the callback. This presents a big problem. Because of this, you can not combine callbacks+closures with exception catching.

Wait.for

...and would change the node landscape

If you use generators to build a helper lib like Wait.for-ES6 (I'm the author), you can completely avoid the callback and the closure, and now "catch blocks" work as expected, and the code is straightforward.

It'd be really helpful if you could give a snippet of code to highlight the difference in common scenarios (making an http request or a db call).

Check Wait.for-ES6 examples, to see the same code with callbacks and with fibers based on generators.

---

UPDATE 2021: All of this has been superseded by javascript/ES2020 async/await. My recommendation is to use Typescript and async/await (which is based on Promises also standardized)

Answer 2

Generators is one of many features in upcoming ES6. So in the future it will be possible to use them in browsers (right now you can play with them in FF).

Generators are constructors for iterators. Sounds like gibberish, so in easier terms they allow to create objects that later will be possible to iterate with something like for loops using .next() method.

Generators are defined in a similar way to functions. Except they have * and yield in them. * is to tell that this is generator, yield is similar to return.

For example this is a generator:

function *seq(){
    var n = 0;
    while (true) yield n++;
}

Then you can use this generator with var s = seq(). But in contrast to a function it will not execute everything and give you a result, it will just instantiate the generator. Only when you will run s.next() the generator will be executed. Here yield is similar to return, but when the yield will run, it will pause the the generator and continues to work on the next expression after next. But when the next s.next() will be called, the generator will resume its execution. In this case it will continue doing while loop forever.

So you can iterate this with

for (var i = 0; i < 5; i++){
  console.log( s.next().value )
}

or with a specific of construct for generators:

for (var n of seq()){
    if (n >=5) break;
    console.log(n);
}

These are basics about generators (you can look at yield*, next(with_params), throw() and other additional constructs). Note that it is about generators in ES6 (so you can do all this in node and in browser).

But how this infinite number sequence has anything to do with callback?

Important thing here is that yield pauses the generator. So imagine you have a very strange system which work this way:

You have database with users and you need to find the name of a user with some ID, then you need to check in your file system the key for a this user's name and then you need to connect to some ftp with user's id and key and do something after connection. (Sounds ridiculous but I want to show nested callbacks).

Previously you would write something like this:

var ID = 1;
database.find({user : ID}, function(userInfo){
    fileSystem.find(userInfo.name, function(key){
        ftp.connect(ID, key, function(o){
            console.log('Finally '+o);
        })
    })
});

Which is callback inside callback inside callback inside callback. Now you can write something like:

function *logic(ID){
  var userInfo  = yield database.find({user : ID});
  var key       = yield fileSystem.find(userInfo.name);
  var o         = yield ftp.connect(ID, key);
  console.log('Finally '+o);
}
var s = logic(1);

And then use it with s.next(); As you see there is no nested callbacks.

Because node heavily uses nested callbacks, this is the reason why the guy was telling that generators can change the landscape of node.

Answer 3

A generator is a combination of two things - an Iterator and an Observer.

Iterator

An iterator is something when invoked returns an iterable which is something you can iterate upon. From ES6 onwards, all collections (Array, Map, Set, WeakMap, WeakSet) conform to the Iterable contract.

A generator(iterator) is a producer. In iteration the consumer PULLs the value from the producer.

Example:

function *gen() { yield 5; yield 6; }
let a = gen();

Whenever you call a.next(), you're essentially pull-ing value from the Iterator and pause the execution at yield. The next time you call a.next(), the execution resumes from the previously paused state.

Observer

A generator is also an observer using which you can send some values back into the generator. Explained better with examples.

function *gen() {
  document.write('<br>observer:', yield 1);
}
var a = gen();
var i = a.next();
while(!i.done) {
  document.write('<br>iterator:', i.value);
  i = a.next(100);
}

Here you can see that yield 1 is used like an expression which evaluates to some value. The value it evaluates to is the value sent as an argument to the a.next function call.

So, for the first time i.value will be the first value yielded (1), and when continuing the iteration to the next state, we send a value back to the generator using a.next(100).

Where can you use this in Node.JS?

Generators are widely used with spawn (from taskJS or co) function, where the function takes in a generator and allows us to write asynchronous code in a synchronous fashion. This does NOT mean that async code is converted to sync code / executed synchronously. It means that we can write code that looks like sync but internally it is still async.

Sync is BLOCKING; Async is WAITING. Writing code that blocks is easy. When PULLing, value appears in the assignment position. When PUSHing, value appears in the argument position of the callback

When you use iterators, you PULL the value from the producer. When you use callbacks, the producer PUSHes the value to the argument position of the callback.

var i = a.next() // PULL
dosomething(..., v => {...}) // PUSH

Here, you pull the value from a.next() and in the second, v => {...} is the callback and a value is PUSHed into the argument position v of the callback function.

Using this pull-push mechanism, we can write async programming like this,

let delay = t => new Promise(r => setTimeout(r, t));
spawn(function*() {
  // wait for 100 ms and send 1
  let x = yield delay(100).then(() => 1);
  console.log(x); // 1

   // wait for 100 ms and send 2
  let y = yield delay(100).then(() => 2);
  console.log(y); // 2
});

So, looking at the above code, we are writing async code that looks like it's blocking (the yield statements wait for 100ms and then continue execution), but it's actually waiting. The pause and resume property of generator allows us to do this amazing trick.

How does it work ?

The spawn function uses yield promise to PULL the promise state from the generator, waits till the promise is resolved, and PUSHes the resolved value back to the generator so it can consume it.

Use it now

So, with generators and spawn function, you can clean up all your async code in NodeJS to look and feel like it's synchronous. This will make debugging easy. Also the code will look neat.

BTW, this is coming to JavaScript natively for ES2017 - as async...await. But you can use them today in ES2015/ES6 and ES2016 using the spawn function defined in the libraries - taskjs, co, or bluebird

Answer 4

Summary:

function* defines a generator function which returns a generator object. The special thing about a generator function is that it doesn't execute when it is called using the () operator. Instead an iterator object is returned.

This iterator contains a next() method. The next() method of the iterator returns an object which contains a value property which contains the yielded value. The second property of the object returned by yield is the done property which is a boolean (which should return true if the generator function is done).

Example:

function* IDgenerator() {
  var index = 0;

  yield index++;
  yield index++;
  yield index++;
  yield index++;
    
}

var gen = IDgenerator(); // generates an iterator object

console.log(gen.next().value); // 0  
console.log(gen.next().value); // 1
console.log(gen.next().value); // 2
console.log(gen.next()); // object, 
console.log(gen.next()); // object done

In this example we first generate an iterator object. On this iterator object we then can call the next() method which allows us to jump form yield to yield value. We are returned an object which has both a value and a done property.

How is this useful?

• Some libraries and frameworks might use this construct to wait for the completion of asynchronous code for example redux-saga
• async await the new syntax which lets you wait for async events uses this under the hood. Knowing how generators work will give you a better understanding of how this construct works.