Why won't yield return from within a `.map` callback?

Question

Learn Generators - 4 » CATCH ERROR! The solution uses a for loop but I just couldn't find anything in MDN - Iteration Protocols that refers to yield within callbacks.

I'm going to guess the answer is just don't do that but thanks in advance if anyone has the time or inclination to provide an explanation!

Code:

function *upper (items) {   items.map(function (item) {     try {       yield item.toUpperCase()     } catch (e) {       yield 'null'     }   } }  var badItems = ['a', 'B', 1, 'c']  for (var item of upper(badItems)) {   console.log(item) } // want to log: A, B, null, C 

Error:

⇒  learn-generators run catch-error-map.js /Users/gyaresu/programming/projects/nodeschool/learn-generators/catch-error-map.js:4       yield item.toUpperCase() // error below             ^^^^ SyntaxError: Unexpected identifier     at exports.runInThisContext (vm.js:73:16)     at Module._compile (module.js:443:25)     at Object.Module._extensions..js (module.js:478:10)     at Module.load (module.js:355:32)     at Function.Module._load (module.js:310:12)     at Function.Module.runMain (module.js:501:10)     at startup (node.js:129:16)     at node.js:814:3 

Even my editor knows this is a terrible idea...

Answer 1

Disclaimer: I'm the author of Learn generators workshopper.

Answer by @slebetman is kinda correct and I also can add more:

Yes, MDN - Iteration Protocol doesn't refer directly about yield within callbacks. But, it tell us about importance from where you yield item, because you can only use yield inside generators. See MDN - Iterables docs to find out more.

@marocchino suggest just fine solution iterate over Array that was changed after map:

function *upper (items) {   yield* items.map(function (item) {     try {       return item.toUpperCase();     } catch (e) {       return null;     }   }); } 

We can do it, because Array has iteration mechanism, see Array.prototype[@@iterator]().

var bad_items = ['a', 'B', 1, 'c'];  for (let item of bad_items) {   console.log(item); // a B 1 c } 

Array.prototype.map doesn't have default iteration behavior, so we couldn't iterate over it.

But generators is not just iterators. Every generator is an iterator, but not vice versa. Generators allows you to customize iteration (and not only) process by calling yield keyword. You can play and see the difference between generators/iterators here:

Demo: babel/repl.

Answer 2

One problem is yield yields just one level to the function's caller. So when you yield in a callback it may not do what you think it does:

// The following yield: function *upper (items) { // <---- does not yield here   items.map(function (item) { // <----- instead it yields here     try {       yield item.toUpperCase()     } catch (e) {       yield 'null'     }   } } 

So in the code above, you have absolutely no access to the yielded value. Array.prototype.map does have access to the yielded value. And if you were the person who wrote the code for .map() you can get that value. But since you're not the person who wrote Array.prototype.map, and since the person who wrote Array.prototype.map doesn't re-yield the yielded value, you don't get access to the yielded value(s) at all (and hopefully they will be all garbage collected).

Can we make it work?

Let's see if we can make yield work in callbacks. We can probably write a function that behaves like .map() for generators:

// WARNING: UNTESTED! function *mapGen (arr,callback) {     for (var i=0; i<arr.length; i++) {         yield callback(arr[i])     } } 

Then you can use it like this:

mapGen(items,function (item) {     yield item.toUpperCase(); }); 

Or if you're brave you can extend Array.prototype:

// WARNING: UNTESTED! Array.prototype.mapGen = function *mapGen (callback) {     for (var i=0; i<this.length; i++) {         yield callback(this[i])     } }; 

We can probably call it like this:

function *upper (items) {   yield* items.mapGen(function * (item) {     try {       yield item.toUpperCase()     } catch (e) {       yield 'null'     }   }) } 

Notice that you need to yield twice. That's because the inner yield returns to mapGen then mapGen will yield that value then you need to yield it in order to return that value from upper.

OK. This sort of works but not quite:

var u = upper(['aaa','bbb','ccc']); console.log(u.next().value); // returns generator object 

Not exactly what we want. But it sort of makes sense since the first yield returns a yield. So we process each yield as a generator object? Lets see:

var u = upper(['aaa','bbb','ccc']); console.log(u.next().value.next().value.next().value); // works console.log(u.next().value.next().value.next().value); // doesn't work 

OK. Let's figure out why the second call doesn't work.

The upper function:

function *upper (items) {   yield* items.mapGen(/*...*/); } 

yields the return value of mapGen(). For now, let's ignore what mapGen does and just think about what yield actually means.

So the first time we call .next() the function is paused here:

function *upper (items) {   yield* items.mapGen(/*...*/); // <----- yields value and paused } 

which is the first console.log(). The second time we call .next() the function call continue at the line after the yield:

function *upper (items) {   yield* items.mapGen(/*...*/);   // <----- function call resumes here } 

which returns (not yield since there's no yield keyword on that line) nothing (undefined).

This is why the second console.log() fails: the *upper() function has run out of objects to yield. Indeed, it only ever yields once so it has only one object to yield - it is a generator that generates only one value.

OK. So we can do it like this:

var u = upper(['aaa','bbb','ccc']); var uu = u.next().value; // the only value that upper will ever return console.log(uu.next().value.next().value); // works console.log(uu.next().value.next().value); // works console.log(uu.next().value.next().value); // works 

Yay! But, if this is the case, how can the innermost yield in the callback work?

Well, if you think carefully you'll realize that the innermost yield in the callback also behaves like the yield in *upper() - it will only ever return one value. But we never use it more than once. That's because the second time we call uu.next() we're not returning the same callback but another callback which in turn will also ever return only one value.

So it works. Or it can be made to work. But it's kind of stupid.

Conclusion:

After all this, the key point to realize about why yield doesn't work the way we expected is that yield pauses code execution and resumes execution on the next line. If there are no more yields then the generator terminates (is .done).

Second point to realize is that callbacks and all those Array methods (.map, .forEach etc.) aren't magical. They're just javascript functions. As such it's a bit of a mistake to think of them as control structures like for or while.

Epilogue

There is a way to make mapGen work cleanly:

function upper (items) {   return items.mapGen(function (item) {     try {       return item.toUpperCase()     } catch (e) {       return 'null'     }   }) } var u = upper(['aaa','bbb','ccc']); console.log(u.next().value); console.log(u.next().value); console.log(u.next().value); 

But you'll notice that in this case we return form the callback (not yield) and we also return form upper. So this case devolves back into a yield inside a for loop which isn't what we're discussing.