Riak Test v1.1.0

[jburwell]

This PR is not intended to be merged at this time. It is being used to review the work done thus far, and begin the process of cleanup for merge into master.

Riak Test v1.1.0 ... more writeup to come.

/cc @javajolt

[jburwell]

Should we remove this script prior to merge? We had always intended to remove it, but it may prove useful until for verifying changes until the v1 wrapper is dropped in v2.0.0.

[hazen]

I think it's still useful for now. Easy to remove later.

[jburwell]

Notes on the planner design

rt_planner and rt_reporter are good starts towards the pipeline design I have in mind for r_t. After reviewing the initial implementation, I have the following observations:

• The GiddyUp dependency can be further isolated in the codebase
• The riak_test_escript module contains more test planning work than necessary
• Isolate the use of rt_config to the riak_test_escript module as much as possible to simplify component unit testing

Ideally, we have a pipeline constructed with the following stages:

planner->scheduler->runner

Each of these stages can emit a test_result event that is broadcast to a list of reporters (e.g. console, giddyup).

A planner stage accepts a list of tests, backends, and groups and creates a set of test plans to be scheduled for execution. The following is (roughly) the test planner API I have in mind:

-spec start_link(pid(), [pid()], proplists:proplist() -> rt_util:result().
start_link(SchedulerPid, ReporterPids, Options) -> %% the options are implementation dependent

-spec plan([atom()], [backend()], [atom()]) -> {ok, non_neg_integer()} | rt_util:error().
plan(Tests, Backends, Groups) ->  %% When the test list is empty, run all available tests

A scheduler stage accepts a test plan, reserves the quantity of nodes requested by the plan, and submits the schedule to an executor stage for execution. The following is (roughly) the test scheduler API I have in mind:

-spec start_link(pid(), [pid()], proplists:proplist()).
start_link(ExecutorPid, ReporterPids, Options) -> %% Options are implementation dependent

-spec schedule(#rt_planner:test_plan()) -> rt_util:result().
schedule(TestPlan) ->

Finally, a Runner stage provisions the nodes allocated in the test schedule, executes the test specified in the test schedule, and reports the results of the test run the to reporters. The following is (roughly) the API I have in mind:

-spec start_link([pids()], proplists:proplist()) -> rt_util:result().
start_link(ReporterPids, Options) ->

-spec run(rt_scheduler:test_schedule) -> rt_util:result().
run(TestSchedule) ->

Based on this (high-level) design, I propose the following refactorings:

1. Convert the giddyup module to a gen_server that has the giddyup configuration information passed into its start_link function. In addition to encapsulating the GiddyUp configuration, it will also allow asynchronous communication with GiddyUp to prevent blocking of the pipeline if/when we have a slow connection.
2. Refine the rt_planner module to define the planner behavior and any common functions used by all planners
3. Move all GiddyUp related planning functions into a new rt_giddyup_planner module -- removing the notion of wrapping test plans in the escript module. If a list of tests, backends, and/or groups is provided to this planner, they will be used to filter the list of test plans retrieved from GiddyUp.
4. Create an rt_simple_planner module which is used when a list of tests is specified on the command line.
5. Add a build_test_pipeline function to the riak_test_escript that selects and starts the planner, scheduler, and reporter(s) and ties them together for execution.

/cc @javajolt