Short test covering most of Riak for development

[engelsanchez]

This is an exploration along the lines of Kelly's proposal: https://gist.github.com/kellymclaughlin/f28b75ac295b3d397ef0

The main goal is to have a single test running in a reasonable amount of time that covers as much of Riak's functionality as possible in that time. Developers can use this before checking in code, since running all existing tests in their current form takes more than a day an destroys your laptop in the process. In order to do this, tests should be refactored to be more modular and reusable, so they can be run in small or large setups. It would be great if they can perform steps in a way that allows different functionality to be mixed. For example, mixing preparing data for 2i test in a bucket with preparing data for a CRDT test, then interleave verification for each. Run the same steps before/after adding/removing nodes, destroying partitions, etc.

The idea is to give up on side effect free runs and favor verifying functionality in the context of other functionality running. This is usually not what you want with unit tests, but the crux of integration testing.

A sketch of possible phases of this test:

• The test creates the smallest cluster possible, with as few vnodes as possible and configured to perform most operations much faster than a real cluster.
• It runs as many functionality verification steps as can run without a setup that makes them incompatible to test with other functionality.
• Perform cluster operations/events such as adding/removing nodes, losing partitions, resizing the ring, etc
• Run verification steps for as much functionality as possible

This is highly experimental and I expect lots of challenges. But the more we can cram into a short running tests, even if not perfect, the more useful this would be day to day.

[kellymclaughlin]

One test to rule them all. I approve.

[reiddraper]

Do we intend for this work to be done before 2.0? Or is it just on the 2.0 milestone for the time being, to see how much we get done?

[engelsanchez]

I consider it part of the finish CI continuum. The more of that we do before 2.1 starts, the better chances we have. And there is plenty of work left in 2.0 that would benefit from test improvements.

[macintux]

@engelsanchez Speaking of dropping items from the RC milestone, this seems like a prime candidate...

[engelsanchez]

Work to ensure we some day have proper CI should continue in parallel. I'm fine with putting in whatever milestone.

[macintux]

Moving to 2.1

[reiddraper]

Work to ensure we some day have proper CI should continue in parallel. I'm fine with putting in whatever milestone.

@engelsanchez +1