Report quartiles

[pkkm]

It would be really useful if Criterion reported quartiles (i.e. the 25th percentile, median and 75th percentile) of the execution time in its terminal output. Unlike the mean and least squares regression, quartiles are fairly insensitive to outliers, so they're good for noisy benchmark data (e.g. due to using a web browser while benchmarking).

[treeowl]

I thought this too. But I'm not sure it actually makes much sense. You'd have to measure CPU time separately for each operation. I don't know that it has a high enough resolution to be meaningful. Furthermore, you'd need to record all those measurements to be able to calculate the quartiles; the variable overhead of that recording seems likely to be a significant source of noise.

[ocramz]

@pkkm @treeowl you could use tdigest for this: http://hackage.haskell.org/package/tdigest