Initialize benchmark in criterion and exclude initialization time from results

Question

I need to benchmark some code inside IO, and criterion supports that pretty well. But I want to perform few initialization steps (different for each benchmark). The naive approach:

main = defaultMain
  [ bench "the first" $ do
      initTheFirst
      theFirst
      cleanUpTheFirst
  , bench "the second" $ do
      initTheSecond
      theSecond
      cleanUpTheSecond
  ]

But it performs initialization and cleanup for every benchmark run (100 times by default) and includes initialization time to final results. Is it possible to exclude initialization time?

ADDED: The code uses global state (mongodb actually), so I can't prepare two initial states simultaneously.

Answer 1

Here is a solution using custom main as suggested by argiopeweb:

import Control.Monad
import Control.Monad.IO.Class
import Control.Concurrent
import Criterion
import Criterion.Config
import Criterion.Monad
import Criterion.Environment

main :: IO ()
main = myMain [
  (initTheFirst, theFirst),
  (initTheSecond, theSecond)
  ]

initTheFirst :: IO ()
initTheFirst = do
  putStrLn "initializing the first"
  threadDelay 1000000

theFirst :: Benchmark
theFirst = bench "the first" $ do
  return () :: IO ()

initTheSecond :: IO ()
initTheSecond = do
  putStrLn "initializing the second"
  threadDelay 1000000

theSecond :: Benchmark
theSecond = bench "the second" $ do
  return () :: IO ()

myMain :: [(IO (), Benchmark)] -> IO ()
myMain benchmarks = withConfig defaultConfig $ do
  env <- measureEnvironment
  forM_ benchmarks $ \(initialize, benchmark) -> do
    liftIO $ initialize
    runAndAnalyse (const True) env benchmark

The output:

warming up
estimating clock resolution...
mean is 1.723574 us (320001 iterations)
found 1888 outliers among 319999 samples (0.6%)
  1321 (0.4%) high severe
estimating cost of a clock call...
mean is 43.45580 ns (13 iterations)
found 2 outliers among 13 samples (15.4%)
  2 (15.4%) high severe
initializing the first

benchmarking the first
mean: 7.782388 ns, lb 7.776217 ns, ub 7.790563 ns, ci 0.950
std dev: 36.01493 ps, lb 29.29834 ps, ub 52.51021 ps, ci 0.950
initializing the second

benchmarking the second
mean: 7.778543 ns, lb 7.773192 ns, ub 7.784518 ns, ci 0.950
std dev: 28.85100 ps, lb 25.59891 ps, ub 32.85481 ps, ci 0.950

You can see that init* functions are called only once and don't affect benchmarks results.

Answer 2

Three real options I can see here. Either initialize and cleanup before and after:

main = do
  initTheFirst
  initTheSecond

  defaultMain
    [ bench "the first" theFirst
    , bench "the second" theSecond
    ]

  cleanUpTheFirst
  cleanUpTheSecond

Or, if that isn't possible, also benchmark your cleanup and initialization processes and modify your benchmark times accordingly.

Alternatively, you could forego using the provided defaultMain and instead roll your own using the lower-level functions provided by Criterion.