TraversableOnce, Future, and Option in a Scala for comprehension

Question

I have a list of string ids representing DB records. I'd like to load them from the DB asynchronously, then upload each record to a remote server asynchronously, then when all are done uploading, make a record of the ids of the records that were uploaded.

Since I'm on Scala 2.9.2, I'm using Twitter's core-util Future implementation, but it should work exactly like the 2.10 futures in terms of Monadic transformations.

The general concept is this:

def fetch(id: String): Future[Option[Record]]
def upload(record: Record): Future[String]
def notifyUploaded(ids: Seq[String]): Unit

val ids: Seq[String] = ....

I'm trying to do this via a for comprehension but the fact that fetch returns a Future of Option makes it obscure and the code doesn't compile:

for {
  id <- ids
  maybeRecord <- fetch(id)
  record <- maybeRecord
  uploadedId <- upload(record)
} yield uploadedId

Compiling this results in the following error:

scala: type mismatch;
found   : com.twitter.util.Future[String]
required: Option[?]
    uploadedId <- upload(record)
                  ^

What am I missing? why does the compiler expect uploadedId to be an Option? is there any pretty way I could work around this?

Answer 1

Consider the signature of the flatMap (or bind) function:

trait Monad[M[_]] {
  def flatMap[A](a : M[A], f : A => M[B]) : M[B]
  ....

In your case, you're trying to use flatMap on an Option, giving it an f that generates a Future. But as in the signature above, f should be generating something in the same monad that it's been called on.

Scala isn't necessarily terribly helpful in this regard, since it's pretty good at converting things around (to Seqs, for example) in such a way that you get the impression that you can chain arbitrary flatMap calls together, regardless of the container.

What you possibly want is a 'Monad transformer', which gives you some ability to compose monads. Debasish Ghosh has a post on using Scalaz monad transformers here.