Dead simple Fork-Join concurrency in Clojure

Question

I have two costly functions that are independent. I want to run them in parallel. I don't want to deal with futures and such (I'm new to Clojure and easily confused).

I'm looking for a simple way to run two functions concurrently. I want it to work like the following

(defn fn1 [input] ...) ; costly
(defn fn2 [input] ...) ; costly

(let [[out1 out2] (conc (fn1 x) (fn2 y))] ...)

I want this to return a vector with a pair of outputs. It should only return once both threads have terminated. Ideally conc should work for any number of inputs. I suspect this is a simple pattern.

Answer 1

Using futures is very easy in Clojure. At any rate, here is an answer that avoids them

(defn conc [& fns]
  (doall (pmap (fn [f] (f)) fns)))

pmap uses futures under the hood. doall will force the sequence to evaluate.

(let [[out1 out2] (conc fn1 fn2)]
        [out1 out2])

Note, that I destructured out1 and out2 in an attempt to preserve your example.

Answer 2

You do need a macro to preserve the desired syntax, though there are other ways of obtaining the same behavior, as the other answers indicate. Here is one way to do it:

(defn f1 [x] (Thread/sleep 500) 5)
(defn f2 [y] 2)

(defmacro conc [& exprs]
  `(map deref
        [~@(for [x# exprs] `(future ~x#))]))

(time (let [[a b] (conc (f1 6) (f2 7))]
       [a b]))
; "Elapsed time: 500.951 msecs"
;= (5 2)

The expansion shows how it works:

(macroexpand-1 '(conc (f1 6) (f2 7)))
;= (clojure.core/map clojure.core/deref [(clojure.core/future (f1 6)) 
;=                                       (clojure.core/future (f2 7))])

You specified two functions but this should work with any number of expressions.

Answer 3

I understand you don't want your final solution to expose futures though it is useful to illustrate how to do this with futures, and then wrap them in something that hides this detail:

core> (defn fn1 [input] (java.lang.Thread/sleep 2000) (inc input))
#'core/fn1                                                                                     
core> (defn fn2 [input] (java.lang.Thread/sleep 3000) (* 2 input))
#'core/fn2                                                                                     
core> (time (let [f1 (future (fn1 4)) f2 (future (fn2 4))] @f1 @f2))
"Elapsed time: 3000.791021 msecs"  

then we can wrap that up in any of the many clojure wrappers around futures. the simplest being just a function which takes two functions and runs them in parallel.

core> (defn conc [fn1 fn2] 
         (let [f1 (future (fn1)) 
               f2 (future (fn2))] [@f1 @f2]))
#'core/conc                                                                                    
core> (time (conc #(fn1 4) #(fn2 4)))
"Elapsed time: 3001.197634 msecs"                                                                          

This avoids the need to write it as a macro by having conc take the function to run instead of the body to evaluate, and then create the functions to pass to it by putting # infront of the calls.

This can also be written with map and future-call:

core> (map deref (map future-call [#(fn1 4) #(fn2 42)]))
(5 84)  

You can then improce conc until it resembles (as Julien Chastang wisely points out) pmap