Why does Scala implement for as a closure?

Question

Recent events on the blogosphere have indicated that a possible performance problem with Scala is its use of closures to implement for.

What are the reasons for this design decision, as opposed to a C or Java-style "primitive for" - that is one which will be turned into a simple loop?

(I'm making a distinction between Java's for and its "foreach" construct here, as the latter involves an implicit Iterator).

More detail, following up from Peter. This bit of Scala:

  object ScratchFor {
    def main(args : Array[String]) : Unit = {
      for (val s <- args) {
        println(s)
      }
    }
  }

creates 3 classes: ScratchFor$$anonfun$main$1.class ScratchFor$.class ScratchFor.class

ScratchFor::main just forwards to the companion object, ScratchFor$.MODULE$::main which spins up an ScratchFor$$anonfun$main$1 (which is an implementation of AbstractFunction1).

It's in the apply() method of this anonymous inner impl of AbstractFunction1 that the actual code lives, which is effectively the loop body.

I don't see HotSpot being able to rewrite this into a simple loop. Happy to be proved wrong on this, though.

Answer 1

Traditional for loops are clumsy, verbose and error-prone. I think it is proof enough of this that "for-each" loops where added to Java, C# and C++, but if you want more details you may check item 46 of Effective Java.

Now, for-each loops are still much faster than Scala for-comprehension, but they are also much less powerful (and more clumsy) because they cannot return values. If you want to transform or filter a collection (or do both to a group of collections), you'll still have to handle all the mechanical details of constructing the result collection in addition to computing the values. Not to mention it inevitably uses some mutable state.

Finally, even though for-each loops are adequate enough for collections, they are not suited to other monadic classes (of which collections are a subset of).

So Scala has a general method which takes care of all of the above. Yes, it is slower, but the goal is to have the compiler effectively optimise it well enough so that this doesn't become a hindrance (and, of course, JIT could help here as well).

That has not been accomplished to this date, but -optimise has reduced a lot of ground between common for-each loops and for-comprehensions on the latest versions of Scala. If performance is essential, you can always use while or tail recursion.

Now, it would be possibly for Scala to have common for loops or for-each loops as special cases specifically targeted at performance issues (since for-comprehensions can do everything they do). However, that violates two principles that guide Scala's design:

1. Reduce complexity. Yes, contrary to what some say, that is a design goal, and special cases that serve no other purpose other than optimise performance -- even though a workable solution exists for performance cases -- would needlessly increase the complexity of the language.

2. Scalability. This is in the sense that the use can scale the language for any size of problem by writing libraries. The point here is that having the compiler optimise one particular class, such as Range, would make it impossible for the user to create a replacement class that would perform just as well.

Answer 2

The for comprehension in Scala is a powerful general-purpose looping and pattern-matching construct. Look at what it can do:

case class Person(first: String, last: String) {}
val people = List(Person("Isaac","Newton"), Person("Michael","Jordan"))
val lastfirst = for (Person(f,l) <- people) yield l+", "+f
for (n <- lastfirst) println(n)

The second case looks pretty straightforward--take each item in a collection and print it. But the first takes apart a list containing a custom data structure and transforms it into a different collection type!

The first for there highlights only a small portion of the capability of the construct; it is both extremely powerful and extremely general. In order to maintain this power, the for must be able to turn into something very general, which means closures. Then the question is: do you also introduce special cases that operate on known collections in simple ways with improved performance? The answer thus far has been mostly no, instead preferring solutions that optimize the general closure-taking methods that for turns into.

Whether this is useful for you in particular depends on whether you are using the general capabilities a lot (in which case you will be glad) or not (in which case you may wish progress was faster).

Still, try -optimize. It often usefully speeds up simple for-comprehensions these days.

Answer 3

The for-comprehension is much more than a simple loop.

If you need an imperative loop, use while. If you want to write performant code in Scala, you need to know this. Just like you have to know about language implementation when you want to write fast code in every other language.

So, since the for-comprehension is not a simple loop, I hope you understand that it's not compiled down to a simple loop.

Answer 4

I would assume using a closure is a general solution. A more optimal solution in some cases would be to "inline" the closure as a loop and eliminate the need to create an object. Perhaps the Scala designers feel the JIT should do this, rather having the compiler do this.

Let's say in Java this is the same as writing

public static void main(String... args) {
    for_loop(args, new Function<String>() {
        public void apply(String s) {
            System.out.println(s);
        }
    });
}

interface Function<T> {
    void apply(T s);
}

public static <T> void for_loop(T... ts, Function<T> tFunc) {
    for(T t: ts) tFunc.apply(t);
}

This is fairly easy to inline (if you're a human). What is surprising is that Scala doesn't have an intrinsic to perform the optimisation to eliminate the need for a new object. Certainly the JIT could do it in theory, but in practise, it might be a while before it handles this specific case.