Ruby's `lazy` less performant than allocating huge lists?

Question

I'm trying to process a large list of numbers:

require 'benchmark'

N = 999999

Benchmark.bm 10 do |bm|
   bm.report 'Eager:' do
 (0..N).select(&:even?).map{|x| x * x}.reduce(&:+)
   end  
   bm.report 'Lazy:' do
 (0..N).lazy.select(&:even?).map{|x| x * x}.reduce(&:+)
   end  
 end;

To my understanding, the lazy version should be much faster, because the eager version needs to allocate two lists of half a million items each(one for select and one for map) while the lazy version is streaming everything.

However, when I run it, the lazy version takes more than twice as long as the eager one! (http://rextester.com/OTEX7399)

         user     system      total        real
Eager:       0.210000   0.010000   0.220000 (  0.216572)
Lazy:        0.580000   0.000000   0.580000 (  0.635091)

How can it be?

Answer 1

I'd say an Enumerator is a far slower middle man than memory is.

This was also reported a while back and Ruby core team member Yusuke Endoh said:

Enumerator::Lazy is not a silver bullet; it removes the overhead for creating an intermediate array, but brings the drawback for calling a block. Unfortunately, the latter is much bigger than the former. Thus, in general, Lazy does bring performance drawback.

---

An analogy I just thought of: Imagine you're building some furniture for a friend.

• Non-lazy: You build the whole thing, rent a truck, and drive it to your friend.

• Lazy: You build a little piece and drive it to your friend with your car. You build the next little piece and drive it to your friend with your car. You build the next little piece and drive it to your friend with your car. And so on.

Yes, renting that truck is extra overhead, but it's nothing compared to driving over and over again with your car.

The real reason that being lazy can save time is that after the first few pieces your friend finds out that you slept with his wife so now you're no longer friends and he doesn't want your stupid furniture anymore and you're not building the remaining pieces at all.