Techniques for wrapping test programs (RunAs, etc.) in v2 and v3

[davidmatson]

I have an xUnit v2 test project that's launched in a different context than the test needs in order to execute successfully (via general-purpose infrastructure that launches everything as SYSTEM, to cover cases where that kind of access is needed, with the option to launch a child process as a user account as appropriate).

Is there a clean way to hook the protocol between the either the VSTS runner and the xUnit DLL, or within the xUnit DLL for test execution, so that it can delegate to another process that will run all the same tests as the current DLL, but in a different context (for example, under a different user account)?

With xUnit v3, I suspect/hope this will be really easy - just re-launch the same test exe in a different environment, preserve the command line arguments, and connect the console output/error (assuming that all communication with the test exe either is based on command-line parameters or via stdout/stderr). Is that an accurate understanding of how v3 would work here, or is the contract between a v3 exe and an outer harness still being defined?

[bradwilson]

[in v2] Is there a clean way to hook the protocol between the either the VSTS runner and the xUnit DLL, or within the xUnit DLL for test execution, so that it can delegate to another process that will run all the same tests as the current DLL, but in a different context (for example, under a different user account)?

There's nothing built in to do this. (I assume this is somewhat related to your question about making your own stand alone executables for v2.)

With xUnit v3, I suspect/hope this will be really easy - just re-launch the same test exe in a different environment, preserve the command line arguments, and connect the console output/error (assuming that all communication with the test exe either is based on command-line parameters or via stdout/stderr). Is that an accurate understanding of how v3 would work here, or is the contract between a v3 exe and an outer harness still being defined?

Certainly, in v3 since the project is itself self-executable, you could run this executable in a different environment so long as you could find a reasonable way to capture the output of the results (for example, launching it to write the results to XML, if you can't observe the console output of the running app). It would be up to you to do the launch/capture the results. Since we don't have our cross process RPC done yet for v3, it's unclear how easy or difficult it might be to influence the actual launching of the process, but if that's a desired feature I'm open to suggestions on how to enable customization of the launch process.

[davidmatson]

Thanks, Brad.

Since we don't have our cross process RPC done yet for v3, it's unclear how easy or difficult it might be to influence the actual launching of the process, but if that's a desired feature I'm open to suggestions on how to enable customization of the launch process.

Yes, it would be great if the v3 protocol made this scenario easy. I think there are two overall levels of support for this scenario:

1. Delegation of Test Execution/Results Reporting
   For example, being able to wrap the entire test exe, perhaps by re-launching it with the same command-line but under a different user's token, and redirecting/capturing console (input?/)output.
2. Aggregation of Test Execution/Results Reporting
   For example, being able to run some tests in the outer exe, delegate to another exe to run some additional tests (again, for example, under a different user account), and then coming back to the outer exe.

The first level is the most important, and probably the easiest to achieve.

The second level is nice-to-have, with some additional pressure put on simplicity of the results reporting contract. (For example, if it's a TCP socket which itself uses a fairly complex, bi-directional protocol, that's quite a bit of plumbing to write. If instead the protocol is unidirectional/output-only, and the boundaries between items are simple to determine, or if it's a stdout line-based format, those are easy to compose together.) It's not critical, but having this design force towards a simple protocol may be useful for other scenarios as well.

A few ideas that could help here:

1. For the launch process, triggering the tests to run and report results programmatically via just executing the exe with a certain command line seems attractive.
   For example, pass no arguments to execute all the tests, pass one extra argument to get machine-readable output (via stdout, a socket, JSON-RPC, etc.). Even having the "what to do" part controlled via the command line would help this scenario (for example, "--list-tests" means to list tests rather than run all of them - something simple like that could make both Delegation and Aggregation scenarios easy to achive.) Other frameworks have used this approach, and it seems to have worked well there.
2. For reporting results, a few interesting options to consider:
   a. Report results as XML, streamed to stdout as the test executes.
   This is what both Boost.Test and Catch2 do. It supports Delegation easily, and probably Aggregation without too much difficulty. Using the stdout stream means it supports callers that need to stream results as the test runs. Reading a partial XML stream is rather complex code to write though, given the state machine-style logic involved (I had to do write streaming parsers for both Catch2 and Boost.Test; it's notably harder than any of the following options.)
   b. Report results via TCP, streamed out as one-line-per-event ("event=TestStart&Name=MyTest&...")
   This is what GoogleTest does. I've been impressed by its elegance - it's easy to write, easy to read/process, human-readable, and supports both Delegation and Aggregation with hardly any effort.
   c. Report results via Test Anything Protocol (which uses stdout).
   I like the idea of a simple, standard format. Again, this option seems easy to write, human-readable, and likely would support Delegation and Aggregation very easily. (I'm not sure about how hard to process/parse; I haven't tried that.) The fact that they're up to v14 makes me a little skeptical though about the overall stability/simplicity.
   d. Report results via stdout, as one-line-per-event (again, perhaps URL QueryString-style: "event=TestStart&Name=MyTest&...")
   This is a minor variation on b - using stdout instead of a separate TCP port means less plumbing code to write; standard output comes "for free".

Any of options b-d could work very well for this scenario (d is especially simple and easy to work with).

Thoughts?