have criterion analyse user supplied timing data?

[erantapaa]

Is it possible to use the criterion analysis and reporting functions on timing data that I supply?

I've tried hacking the criterion code to get this simple example to work:

{-# LANGUAGE RecordWildCards #-}

import Criterion.Monad (withConfig)
import Criterion.Main (defaultConfig)
import Control.Monad.Except
import Criterion.Internal
import Criterion.Types
import qualified Data.Vector as V

theData = [ 2.0, 2.1, 2.2, 2.1, 2.1, 2.15, 2.09 ] :: [Double]

mkMeas t = Measured t t cycles 1 0 0 0 0 0 0 0 
  where cycles = floor (1000000*t)

main = do
  let ms = map mkMeas theData
  withConfig defaultConfig $ do
    r <- analyseOne 1 "foo" (V.fromList ms)
    liftIO $ putStrLn "done"

The changes I made to the criterion source to get this to "work" were:

• export analyseOne from Criterion.Internal
• in analyseSample set overhead to a constant (0.00001)

However, this reports a negative value for "time":

time                 -9.4833e15 s (-9.6763e15 s .. -9.2838e15 s)
                     -3.497 R²  (-9782.939 R² .. -1.158 R²)
mean                 2.123 s    (2.097 s .. 2.175 s)
std dev              43.20 ms   (21.60 ms .. 58.54 ms)
variance introduced by outliers: 12% (moderately inflated)

Am I on the right track, or am I missing something? Thanks!

[rrnewton]

I don't know the code well enough to comment yet about whether this is the right track.

I will say that us at IU want to be able to use criterion with measurement data other than realtime. For example, often benchmarks (especially those that execute in a subprocess) can report their own time more accurately, and we don't yet have a way to communicate this to criterion.

For example, the hsbencher framework has a convention where a benchmark that dumps "SELFTIMED: 3.3s" to stdout thereby overrides the wallclock time that would have been assigned to the benchmark. That's the kind of thing we were thinking of.

But this topic is related... it seems both would be addressed by some more general way of intervening in the measure-analyze-report pipeline.