Microbenchmark Clojure functions

Question

Question

How fast are small Clojure functions like assoc? I suspect that assoc operates in the 100ns to 3us range, which makes it difficult to time.

Using time

user=> (def d {1 1, 2 2})
#'user/d
user=> (time (assoc d 3 3))
"Elapsed time: 0.04989 msecs"
{1 1, 2 2, 3 3}

There is clearly a lot of overhead there so I don't trust this benchmark. Friends pointed me to Criterium which handles a lot of the pain of benchmarking (multiple evaluations, warming up the JVM, GC see How to benchmark functions in Clojure?).

Using Criterium

Sadly, on such a small benchmark even Criterium seems to fail

user=> (use 'criterium.core)
nil
user=> (def d {1 1 2 2})
#'user/d
user=> (bench (assoc d 3 3))
WARNING: JVM argument TieredStopAtLevel=1 is active, and may lead to unexpected results as JIT C2 compiler may not be active. See http://www.slideshare.net/CharlesNutter/javaone-2012-jvm-jit-for-dummies.
WARNING: Final GC required 1.694448681330372 % of runtime
Evaluation count : 218293620 in 60 samples of 3638227 calls.
             Execution time mean : -15.677491 ns
    Execution time std-deviation : 6.093770 ns
   Execution time lower quantile : -20.504699 ns ( 2.5%)
   Execution time upper quantile : 1.430632 ns (97.5%)
                   Overhead used : 123.496848 ns

Just in case you missed it, this operation takes -15ns on average. I know that Clojure is pretty magical, but negative runtimes seem a bit too good to be true.

Repeat the Question

So really, how long does an assoc take? How can I benchmark micro operations in Clojure?

Answer 1

Why not just wrap it in a loop and renormalize?

On my hardware,

(bench (dotimes [_ 1000] (assoc d 3 3)))

yields a mean execution time of roughly 1000x that of

(bench (assoc d 3 3))

namely, about 100 µs in the first case, and 100 ns in the second. If your single-assoc is "in the noise" for Criterium, you could try wrapping it in the same way and get pretty close to the "intrinsic" value. ((dotimes [_ 1000] 1) clocks in at .59 µs, so the extra overhead imposed by the loop itself is comparatively small.)

Answer 2

Criterium tries to net out the overhead due to its own measurement. This can result in negative results for fast functions. See the Measurement Overhead Estimation section of the readme. Your overhead is suspiciously high. You might run (estimatated-overhead!) [sic] a few times to sample for a more accurate overhead figure.