Enhanced 'for' loop and lambda expressions

Question

To my understanding, lambda expressions capture values, not the variables. For example, the following is a compile-time error:

for (int k = 0; k < 10; k++) {
    new Thread(() -> System.out.println(k)).start();
    // Error—cannot capture k
    // Local variable k defined in an enclosing scope must be final or effectively final
   }

However when I try to run the same logic with enhanced for-loop everything is working fine:

List<Integer> listOfInt = new Arrays.asList(1, 2, 3);

for (Integer arg : listOfInt) {
    new Thread(() -> System.out.println(arg)).start();
    // OK to capture 'arg'
 }

Why is it working fine for an enhanced for loop and not for a normal conventional for loop, although the enhanced for loop is also somewhere inside incrementing the variable as done by a normal loop.**

Answer 1

Lambda expressions work like callbacks. The moment they are passed in the code, they 'store' any external values (or references) they require to operate (as if these values were passed as arguments in a function call. This is just hidden from the developer). In your first example, you could work around the problem by storing k to a separate variable, like d:

for (int k = 0; k < 10; k++) {
    final int d = k
    new Thread(() -> System.out.println(d)).start();
}

Effectively final means, that in the above example, you can leave the 'final' keyword out, because d is effectively final, since it is never changed within its scope.

for loops operate differently. They are iterative code (as opposed to a callback). They work within their respective scope and can use all variables on their own stack. This means, that the for loop's code block is part of the external code block.

As to your highlighted question:

An enhanced for loop does not operate with a regular index-counter, at least not directly. Enhanced for loops (over non-arrays) create a hidden Iterator. You can test this the following way:

Collection<String> mySet = new HashSet<>();
mySet.addAll(Arrays.asList("A", "B", "C"));
for (String myString : mySet) {
    if (myString.equals("B")) {
        mySet.remove(myString);
    }
}

The above example will cause a ConcurrentModificationException. This is due to the iterator noticing that the underlying collection has changed during the execution. However in your very example, the external loop creates an 'effectively final' variable arg which can be referenced within the lambda expression, because the value is captured at execution time.

The prevention of the capture of 'non-effectively-final' values is more or less just a precaution in Java, because in other languages (like JavaScript e.g.) this works differently.

So the compiler could theoretically translate your code, capture the value, and continue, but it would have to store that value differently, and you would probably get unexpected results. Therefore the team developing lambdas for Java 8 correctly excluded this scenario, by preventing it with an exception.

If you ever need to change values of external variables within lambda expressions, you can either declare a one-element array:

String[] myStringRef = { "before" };
someCallingMethod(() -> myStringRef[0] = "after" );
System.out.println(myStringRef[0]);

Or use AtomicReference<T> to make it thread-safe. However with your example, this would probably return "before" since the callback would most likely be executed after the execution of println.

Answer 2

In an enhanced for loop the variable is initialized every iteration. From §14.14.2 of the Java Language Specification (JLS):

...

When an enhanced for statement is executed, the local variable is initialized, on each iteration of the loop, to successive elements of the array or Iterable produced by the expression. The precise meaning of the enhanced for statement is given by translation into a basic for statement, as follows:

• If the type of Expression is a subtype of Iterable, then the translation is as follows.

  If the type of Expression is a subtype of Iterable<X> for some type argument X, then let I be the type java.util.Iterator<X>; otherwise, let I be the raw type java.util.Iterator.

  The enhanced for statement is equivalent to a basic for statement of the form:

  for (I #i = Expression.iterator(); #i.hasNext(); ) {
      {VariableModifier} TargetType Identifier =
          (TargetType) #i.next();
      Statement
  }
  

...

• Otherwise, the Expression necessarily has an array type, T[].

  Let L1 ... Lm be the (possibly empty) sequence of labels immediately preceding the enhanced for statement.

  The enhanced for statement is equivalent to a basic for statement of the form:

  T[] #a = Expression;
  L1: L2: ... Lm:
  for (int #i = 0; #i < #a.length; #i++) {
      {VariableModifier} TargetType Identifier = #a[#i];
      Statement
  }
  

...

In other words, your enhanced for loop is equivalent to:

ArrayList<Integer> listOfInt = new ArrayList<>();
// add elements...

for (Iterator<Integer> itr = listOfInt.iterator(); itr.hasNext(); ) {
    Integer arg = itr.next();
    new Thread(() -> System.out.println(arg)).start();
}

Since the variable is initialized each iteration it is effectively final (unless you modify the variable inside the loop).

In contrast, the variable in the basic for loop (k in your case) is initialized once and updated each iteration (if a "ForUpdate" is present, e.g. k++). See §14.14.1 of the JLS for more information. Since the variable is updated each iteration is is not final nor effectively final.

The need for a final or effectively final variable is mandated and explained by §15.27.2 of the JLS:

...

Any local variable, formal parameter, or exception parameter used but not declared in a lambda expression must either be declared final or be effectively final (§4.12.4), or a compile-time error occurs where the use is attempted.

Any local variable used but not declared in a lambda body must be definitely assigned (§16 (Definite Assignment)) before the lambda body, or a compile-time error occurs.

_{Similar rules on variable use apply in the body of an inner class (§8.1.3). The restriction to effectively final variables prohibits access to dynamically-changing local variables, whose capture would likely introduce concurrency problems. Compared to the final restriction, it reduces the clerical burden on programmers.}

_{The restriction to effectively final variables includes standard loop variables, but not enhanced-for loop variables, which are treated as distinct for each iteration of the loop (§14.14.2).}

...

That last sentence even explicitly mentions the difference between basic for loop variables and enhanced for loop variables.

Answer 3

The other replies are helpful, but they don't seem to tackle the question directly and answer it in clear terms.

In your first example, you're trying to access k from the lambda expression. The problem here is that k changes its value over time (k++ is called after each loop iteration). Lambda expressions do capture external references, but they need to be marked as final or be "effectively final" (i.e., marking them as final would still produce valid code). This is to prevent concurrency problems; by the time the thread you created is run, k could already hold a new value.

In your second example, on the other hand, the variable you're accessing is arg, which is reinitialized with every iteration of the enhanced for-loop (compare with the example above, where k was merely updated), so you're creating an entirely new variable with each iteration. As an aside, you can also explicitly declare the iteration variable of an enhanced for-loop as final:

for (final Integer arg : listOfInt) {
    new Thread(() -> System.out.println(arg)).start();
}

This ensures that the value arg references won't have changed by the time the thread you created is run.

Answer 4

An enhanced for loop is defined to be equivalent to this code:

for (Iterator<T> it = iterable.iterator(); it.hasNext(); ) {
    T loopvar = it.next();
    …
}

This substitution code explains why the variable of an enhanced for loop is considered effectively final.