How to perform distributed computations with F# (.Net)

Question

So I just had a little fun in F# with the Async library recently and was very amazed by especially Async.Parallel which basically takes a sequence of Async tasks and unifies them under one Async task.

However, now I am curious on how to distribute a computation task between multiple computers, say for example the two laptops on my desk.

Is it somehow possible to serialize these Async tasks and send them to another computer which then performs the task and sends the result back?

Or maybe I need to serialize the data itself and send it to the other computer, on which I have some code running that performs the computation and sends result back?

Or maybe there is another simplistic way to do it?

What is the general approach to distributed computing in F# using .Net? (recommended design patterns, tools, libraries, etc.)

My end goal is to split a big computation task into smaller parts and run them on multiple machines. Preferably in a simplistic non-enterprise-overly-complex-way.

Answer 1

There is a project called MBrace which does pretty much exactly what you describe :-).

It lets you write cloud computations using the cloud block:

let first = cloud { return 15 }
let second = cloud { return 27 }

You can compose them using let! as with asynchronous workflows and you can also create them from asynchronous workflows using Cloud.ofAsync. Cloud computations can be distributed over network using Cloud.Parallel:

cloud {
  let! results = [ first; second ] |> Cloud.Parallel
  return List.sum results }

Currently, there are MBrace bindings for running the computation locally (for testing) and inside Azure cluster, but there is some work-in-progress on supporting Amazon too.

For more information see mbrace.io and there is also a nice talk from Mathias Brandewinder on crunching through big data with MBrace