WIP: docs: Add high-level QA framework

[thanethomson]

Closes #9020

📖 Rendered

@josef-widder and @sergio-mena, this is my first attempt at trying to codify Josef's QA framework in a way that's relevant to Tendermint.

Where I need some assistance at present is in answering the following questions and filling in some missing information:

• Have we adequately captured all of the high-level concerns?
• Have we captured all logical protocols that Tendermint implements?
• Have we captured all protocol-specific concerns? (i.e. are all of the tables complete for each logical protocol?)
• @josef-widder does the QA Log section adequately address the "correctness log" idea that you mentioned to me?

What's left to be done here before this PR can be considered "ready for review":

• Fill out the remaining specific concerns for all high-level concerns
• Fill out the tables indicating the level of completion for all relevant approaches for each concern
• Fill out the tables with the level of confidence that each approach gives relative to the specific concern

---

PR checklist

• Tests written/updated, or no tests needed
• CHANGELOG_PENDING.md updated, or no changelog entry needed
• Updated relevant documentation (docs/) and code comments, or no
  documentation updates needed

[josef-widder]

These two are quite specific for ABCI++. I would replace them in all tables by "Implementation ensures safety and liveness as given in the specification". Or perhaps something more catchy.

[sergio-mena]

I have the impression that this suggestion can be merged with "Robustness" below

[tychoish]

I don't know that this view of the problem leads to an effective prioritization of methodology, or gives us particular confidence in any of these dimensions.

While I think the best approach to this would be to start from stability/operational/performance requirements and work backwards from that, I think it'd be useful to frame the components that are the highest priority (e.g. memory leaks in testnet/e2e testing, error handling in static analysis, etc.) rather than divide our time attempting to cover everything at every level.

[josef-widder]

The point is not to cover every concern with every method but to eventually figure out what is the most favorable methods for what concern.

In the beginning, let's just record what methods we are using and what concerns we believe we can address with them.

Eventually, we should also identify bottlenecks. E.g., if the only method we are using to find memory leaks are testnets (which seem super expensive), it might make sense to figure out whether a different (cheaper) methods will give us similar confidence. Such an overview allows us to identify such bottlenecks.

[thanethomson]

While I think the best approach to this would be to start from stability/operational/performance requirements and work backwards from that

This is implicit in the correctness evaluation for all of the specific protocols that Tendermint implements.

[sergio-mena]

Thanks @thanethomson for this.

[sergio-mena]

Consider adding an entry here with something like "Protocol/Interfate Specification" (setup cost high, "per-execution cost" empty). Or, if you don't consider it an "approach", this info should be somewhere in this document.

This is needed for, at least, MBT and protocol audits mentioned above. Although, theoretically, it would also be needed to be able to say is a test (at whatever level) is passing or not.

[sergio-mena]

Please add a field to this template with the outcome: "PASS, FAIL, INCONCLUSIVE".
The "INCONCLUSIVE" field is typically used when you try to run a test case (e.g. using a complex testnet script), and the test cannot be run because of problems outside the scope of the UUT.

[josef-widder]

I like the "outcome" idea. I am not sure whether we should give a list to choose from. For instance, if the item describes an audit, then we could say "We opened the following issues"

[sergio-mena]

I have the impression that this suggestion can be merged with "Robustness" below

[sergio-mena]

Clarifying question: what is the "type" of the cells? Is it a boolean to say we're already doing it? Is it something like {"N/A", "planned", "done", "in progress", "nice to have"}?