Find a way to restore or replace cimetrics

[eddyashton]

We disabled cimetrics in #6125, to avoid use of METRICS_MONGO_CONNECTION.

We should look at restoring cimetrics, either with improved auth to mongo, or by finding an alternative storage mechanism.

[achamayou]

I looks like bencher.dev may be what we want.

Test job: https://github.com/microsoft/CCF/blob/bencher_experiment/.github/workflows/bencher.yml, works on ubuntu 20.04 using a 1ES pool.

The json adapter supports both latency and throughput, and the format is very simple: https://bencher.dev/docs/explanation/adapters/#-json

I think we would move the performance tests to GHA for simplicity, and we would start with just running on main. The setup for PRs on branches seems to need a bit of care to avoid exposing secrets, but we may be able to do without it and just check periodically.

[achamayou]

pi_basic_virtual_cft

• Don't need the ^ anymore
• Can we create a Memory slug for the _mem?
• Move other end to end tests
• use high low values?
• Micro-benchmarks?

[achamayou]

Everything has been moved over now. I think there are probably two last things to explore before closing this task:

• Rename the metrics to something more sensible.
• Split the micro-benchmarks into a separate slug, since they're on a fundamentally different scale from end to end. This could still be quite generic, e.g. Rate.

Proposed renames:

• CHAMP get (/s) -> CHAMP get
• CHAMP put (/s) -> CHAMP put
• KV deser (/s) -> KV deserialisation
• KV ser (/s) -> KV serialisation
• KV snap deser (/s) -> KV snapshot deserialisation
• KV snap ser (/s) -> KV snapshot serialisation
• RB get (/s) -> RBMap get
• RB put (/s) -> RBMap put
• commit_latency_ratio -> Commit
  • This one I think we ideally move away from ratio to real latency, and split into multiple runs by signature interval, e.g. Commit (100ms sig_ms_interval)
• historical_queries -> Historical Queries
• pi_basic_js_virtual -> Basic JS
• pi_basic_mt_virtual -> Basic Multi-threaded
• pi_basic_virtual -> Basic
• pi_ls_jwt_virtual -> Logging JWT
• pi_ls_virtual -> Logging