flatMap(func) versus flatMap(func(_))

Question

I'm very surprised by not being able to find an existing question for that one. Why is that, given:

val p: Int => Option[Int] = Some(_)
List(1, 2, 3).flatMap(p)

I'm getting:

<console>:14: error: type mismatch;
 found   : Int => Option[Int]
 required: Int => scala.collection.GenTraversableOnce[?]
       List(1, 2, 3).flatMap(p)

But if I replace last line with this one, it compiles and works as expected:

List(1, 2, 3).flatMap(p(_))

My take on the problem is that in case of p(_) the type inference system kicks in to decide on the type of lambda, and on the way it finds appropriate implicit conversion for Option[Int] (option2Iterable, I believe). With just p, the type is already known, and it's incorrect, so no conversion is attempted (and there's no conversion for Function1 returning Option to Function1 returning GenTraversableOnce).

Is this reasoning right? And if so, is there some reason why I shouldn't report this as a bug/issue?

EDIT: A new twist: I've seen p.apply mentioned in some (sadly) deleted comment (though this was about coding style). Surprisingly, it works just as well as p(_) does.

Answer 1

When you type List(1, 2, 3).flatMap(p(_)) what's done behind the scenes is that function p gets spawned and wrapped in another function that partially applies it - meaning all necessary implicit conversions, if any, will also get applied inside the body of this new function.

When you type List(1, 2, 3).flatMap(p), no function application happens, and you try to pass an Int => Option[Int] which is incompatible with the signature Int => GenTraversableOnce[Int], and although the scope contains an implicit conversion from Option[T] to Iterable[T], there's no conversion from Function1[Int, Option[Int]] to Function1[Int, Iterable[Int]] defined.

The reason for that, probably, is because functions of arbitrary arity have virtually infinite amount of variations due to generics, and since Functions do not share a supertrait, that would require quite a bunch of implicits for each type of functions.

---

Here's a construct that extends flatMap just enough to achieve the desired result for p. However, it makes the already obscure signature of flatMap even less clear (much less clear). I believe, there's no technical block from implementing this behavior, but complexity of signatures is the reason why scala-collections library is often hailed.

import scala.collection.GenTraversableOnce
import scala.collection.generic.CanBuildFrom

implicit class ListEx[A](list: List[A]) {
  def flatMap2[B, M[_], That](f: A => M[B])
                             (implicit bf: CanBuildFrom[List[A], B, That],
                                       view: M[B] => GenTraversableOnce[B]): That =
    list.flatMap(f andThen view)
}

val p: Int => Option[Int] = Some(_)

List(1, 2, 3) flatMap2 p

Answer 2

List(1, 2, 3).flatMap(p(_))

is compiled to:

List(1,2,3).flatMap(x => p(x))

And as p(x) is returning Option[Int] and flatMap needs GenTraversableOnce[Int] so scala.Option.option2Iterable is applied.

Option does not inherit from GenTraversableOnce. To make this syntax work:

List(1,2,3).flatMap(p)

you need implicit convertion from Int => Option[Int] to Int => GenTraversableOnce[Int], sth like this:

import scala.collection.GenTraversableOnce

implicit def conv(c: Int => Option[Int]): Int => GenTraversableOnce[Int] = {
    a => Option.option2Iterable(c(a))
}
val p: Int => Option[Int] = Some(_)
List(1, 2, 3).flatMap(p)

For me this is not a bug, but I agree, it's not intuitive either.