Add runner support for v3 cross-process RPC #2337
Comments
|
Is it intended for the runner API to also use this mechanism? I ask as it will be difficult for us to implement integration with xunit v3 if the test execution happens in a sub-process outside of the outer runner's control. Ideally, we'd want to work with xunit v3 in the same basic manner as v2 (i.e. invoking the v3 runner utility as a service which calls the test code inside the current process). I accept the reasoning behind having the test code inside the user's own EXE process as this definitely solves issues with cross compatibility of so many platform versions. However, in the case of NCrunch (and likely other runners), we've already had to solve these problems. Calling into the runner directly would help with leveraging our existing infrastructure. |
|
@remcomulder wrote:
Yes, this is the primary way it will be surfaced is through the runner API.
Our current thought here is that the runner API will expose the notion that, when a sub-process has been launched, that process ID will be made available to the runner. At the moment, this is planned only for v3, but there is a possibility that for better performance and isolation, we may end up also supporting v2 projects running in a separate process (instead of just a separate app domain). The notification here is critical for things like being able to attach debuggers or profilers.
Running in-process will not be an option for v3. I'm interested to know what other facilities you might need beyond just process ID that we could make available in the runner API.
It's just not possible for us to do this. The assembly loading and linking resolution issues are simply too complex for us to maintain; we tried in v2 with our own .NET Core-based console runner, and it was frankly a complete disaster. It also caused us to not be able to take any third party dependencies outside of the core framework itself; not even dependencies from Microsoft-owned projects that were imported via NuGet. |
Unfortunately, this is not a simple question for me to answer directly. We've stacked a 14 year long project on the ability to have complete control over the environment in which a test is being executed. There's a ton of stuff we've built that I know won't work without this control, and probably at least as much again that I can't account for due to the scale of the change. It's a big paradigm shift for us and it comes with a fair share of unknown unknowns. Because we run user code continuously in transiently wired environments, our reliability and performance constraints go far beyond those you would need to cater for in the standard xunit runners. If the xunit API takes responsibility for the process and environment, these constraints of ours get pushed down onto xunit. I doubt I could convince you to care about them and we won't be able to provide our current features without them being addressed.
I understand the nightmare that comes with needing to host .NET Core (we have some serious war scars from this ourselves). For what it's worth, I think that your chosen architecture makes sense and I think you should absolutely do it. But what about providing a switch that allows a runner to instruct xunit to simply call directly into test execution instead of isolating this in a sub-process? To me it seems like a small abstraction could enable us to both have what we want here. I expect that other runners will probably want to have this option too. A situation that I'm trying to avoid is one where we're forced to write our own xunit emulator because we can't integrate with the API in a way that gives our users the experience they can reasonably expect from NCrunch. Such an approach would feel like a tragic lost opportunity and would also dilute the value of work done to xunit itself. We had to do this with MSTest. I really don't want to do it again. |
|
I really like the v3 choice of making tests projects compile as programs/exes - I've worked on integrating a number of test frameworks into an existing, complex, internal testing infrastructure, including GoogleTest, Catch2, and Boost.Test, and they also compile to exes and their "just run the exe to execute the tests" simplicity and ease launching/debugging/etc. is hard to beat. Two key properties that have made integrating with existing testing infrastructure feasible for those cases has been:
For v3 RPC, have there been further thoughts on exactly what the RPC contract would be, and how it would be triggered from outside the test program? |
|
Another angle to consider here is in designing the RPC contract is how easy it is to nest/delegate test execution. This discussion has one example of where that can provide elegant solutions for problems that are otherwise very challenging. Stated another way, a desirable property here would be that the contract between the test program and any outside runner is something that's easy to pass across multiple process boundaries, such as:
If the protocol is also easy to "proxy" (add some output data from this process, then pass back anything coming from another process), that's a plus. |
|
I've been discussing reusing the new VSTest protocol for our RPC. It's based on JSON-RPC, sometimes over TCP and sometimes over a named pipe. |
|
It's very early, and as such the system is under-documented. https://github.com/microsoft/testfx/blob/main/docs/mstest-runner-protocol/001-protocol-intro.md |
|
Interesting - yeah, I wondered if that would be a good option. It might also be worth considering whether the command line + process output could serve the same purpose. That's the approach the major C++ testing frameworks seem to take (GoogleTest, Catch2, Boost.Test), and it seemes to work well there. For example, to list tests, just pass that command-line switch; to execute tests, just run the exe; if you want to run the exe under a debugger, do so, just like with any other exe; if you want to run a single test, pass the filter switch on the command line, etc. Since the exe command line, and output are needed anyway, re-using them for discover/execution/reporting could be a really simple solution, and it means any harness doesn't even need to have something like a JSON-RPC parser to make use of the exe - even a human can do it just by typing a command, running a PowerShell script, etc. The only place I notice in the new MSTest protocol that looked like more that what the frameworks above handle would be attachments, though I'm not sure if xUnit does those anyway, and those could just be files written to the working directory, or something like that. Thoughts? |
|
I had thought about using the console instead of TCP, it just seemed like TCP would offer me a fairly transparent way to support remoting (the argument could just be a port, but could also be a hostname) and/or container-based testing. It certainly sounds like they're going to be supporting a variety of communication mechanisms (as previously mentioned, at least TCP and named pipes), so I'm in a little bit of a holding pattern waiting for them to start to formalize things a little better. What they have right now is pretty hacked together and un(der)-documented, so I had to decompile some of their closed source packages in order to make anything work, but I did at least get integration working with their |
I'm curious to hear more - what kind of remoting and container scenarios did you have in mind? I tend to think of "run an exe and get its stdout/stderr" as a container-based thing as well, but maybe you're thinking more of a long-lived "test server" process that you'd communicate with from outside the container/across a network boundary, or something like that? I wonder if the core "support machine-triggerable execution and machine-readable results" should be thought of as the same thing or a different thing than cross-process RPC? For example, if the exe supports a parent process triggering it to list tests, execute tests (with an optional filter) and report back the results, does that cover all the goals of a cross-process RPC mechanism? Another goal here might be ensuring we keep a common contract/plugin model for all .NET test frameworks, serving as a replacement for the VSTest layers today; for example, giving support for things like writing TRX files (if the xUnit exe doesn't write to that format natively) or custom test result formats, plugins, etc. Maybe test framework adapters could run in-proc with a meta-runner though? I like how much simpler the MSTest team's protocol is than the current VSTest one, but having the core contract be just .NET types running in-proc with the meta-runner, rather than an RPC wire format, might simplify things even further and avoid any base overhead (for example, when running many C++ unit test exes, if called with the appropriate parameters and output transformed appropriately; an adapter is needed, but not necessarily another process or RPC or new wire format). What kind of goals were you thinking of for an RPC mechanism, and could the console app contract itself (command line, stdin/stdout/stderr) be a good fit for that purpose? |
It's definitely two way. The simplest scenario would be, for example, launching the test process, asking it to enumerate the tests, and then hand back a subset of those tests to be run. This is effectively what all third party test runners do, although in the case of VSTest they do it through two separate processes (for historical reasons related to devices). Hence the belief that this is an RPC mechanism. As for "machine readable results" we already do that with several of our reporters; some are automatic (for example, supporting TeamCity or VSTS/Azure Pipelines), and some are manual (for example, specifying the JSON reporter so that each message is reported as a stand-alone JSON object on the console). Comparing normal output:
To JSON output:
(I did just notice a couple of bugs with
The reality is that that's what VSTest is trying to build right now again. They have always been the common model which all test frameworks support, if they want to be able to integrate into Visual Studio, VS Code,
That is how the current model works. For us that adapter lives in
The problem is: we need processes. The plug-in model is fundamentally broken when considering the complexities of dependency resolution. Real dependency resolution only happens when the compiler builds an executable. The VSTest team has basically had to re-implement the dependency resolution logic as though they were a compiler. It also brings along a whole host of inconveniences: we can't take any third party dependency in our meta-runner that isn't part of the .NET redistributable. An example of where this bit us was trying to use JSON.NET, and then the author of JSON.NET wants to test JSON.NET, and of course we're targeting different versions, and since I'm the executable, my version wins. This was offset (mostly) in .NET Framework by using app domains, but that's not an option any more post .NET Framework. And the dependency resolution logic is VERY non-trivial to implement when you consider that it's not just managed dependencies. This is why we made, and then subsequently gave up on, a .NET Core-based meta-runner. Our meta-runners right now are .NET Framework only, because our answer to dependency resolution problems is app domains. You can turn off app domains but if things fail, then you're stuck, so we turn them on by default and assume you're on your own if you want to turn them off.
I don't have a full implementation laid out yet, but I keep a branch where I've done some experimental work. I have a runner engine that lives in the meta-runner, and an execution engine that lives inside the test project. The "protocol" at the moment is just a simple one-command-per-line system. It's not even remotely fully implemented yet, just enough to start playing around and see what works/doesn't work. The runner engine is quite simple: it receives an INFO message as a startup handshake, and then receives MSG messages which are basically the JSON encoded version of the v3 object model status messages (something like you see above, but more complex). The execution engine is more complex: it also receives an INFO message for the startup handshake, and then receives commands like FIND (for discovery), RUN (for execution), CANCEL, and QUIT. The eventual complexity comes in the options that would available for both FIND and RUN. I'm unsure at this point what other commands would be necessary, as I'm trying to keep it fairly lightweight. The code is old at this point; I've been trying to keep up with changes in |
I guess I'm wondering what parts of this functionality go in the test exe itself vs. a meta-runner. If the exe itself has at least one machine-readable output format, that means, in theory, the meta-runner can be built on top of that format, and the same meta-runner can be used with any exe that writes that format. (I'm using the "produce compatible output" technique for a related integration with an xUnit v2 exe and existing test infrastructure.) The existing JSON output format of xunit.console.exe could be nice.
FWIW, I thought it was the MSTest team rather than the VSTest team behind the new exe runner (the new runner lives in the MSTest repo; the current runner lives in the VSTest repo). But maybe there isn't really a difference between the "VSTest" and "MSTest" teams anymore and it's just the same folks that do both?
Yeah, I wonder if that's a good reason to keep an actual xUnit exe decoupled from any specific meta-runner's protocol - if you want to see results via VS, VSCode, dotnet test, etc., you'll get least common denominator, but if you just run the exe you aren't limited that way, and can do direct integration with the xUnit exe rather than the meta-runner if you want better fidelity for any given scenario.
The point about dependencies is a key one, and I want to make sure I'm understanding it - is the dependency problem here within the test exe (how xUnit, apart from any meta-runner, allows testing code such as Json.NET) or between the meta-runner and the test exe (i.e., in the test adapter code)? In both cases, if any dependencies outside the BCL are avoided, does that help or solve the problem? (For example, if the test adapter is an in-process DLL, but it doesn't need anything apart from BCL because the protocol with the exe is so simple that it's just lauch a process and run a very simple parser on output, does that resolve the problem here?) My guess is that, within the test EXE itself, having xUnit not have any dependency other the BCL and core xUnit packages would be key, but maybe you're imagining another process hop outside the test EXE to run the test code in a more dependency-free environment, or something like that? (And, if that "BCL dependencies only" approach works for the test EXE itself, where test code runs in-proc with xUnit code in the compiled test exe, would it also work for the test adapter being in-proc with a meta-runner? Or maybe having the in-proc stuff all be compiled together is a key reason it doesn't work that way in practice.) |
For the purposes of discussion/architecture, the behavior would need to be built into the This might be able to work. I would have to think about all the operations that would need to be performed and see how easy it would be to ensure they all took place, and were all available as command line switches that could be passed along. Unfortunately it would not be able to support protocol negotiation; you would have to assume a fixed set of available command line switches were available, exactly that that shipped with the first iteration of v3 and nothing else. That would be a potentially significant limitation. The advantage to using a legitimate bi-directional protocol is that negotiation phase where the meta-runner can say "hey, I support v3 of the protocol" and the test project says "I support v2 of the protocol" so the meta-runner (in reality, the runner utility library) then knows that kinds of things it can and can't ask of the test project.
I honestly don't know, but I suspect they're very closely aligned if not the same team.
The problem is specifically resolving the dependencies of the test project. When the test project is a DLL, then the compiler-driven dependency resolution only happened when we (the xUnit.net team) compiled the console runner executable, and only for its dependencies. The dependency resolution for the test project itself has to be done at runtime. We are already forced to subscribe to dependency failures (via the AssemblyHelper for .NET Framework or .NET Core) so that we can help that dependency resolution process. The version for .NET Core is not used anywhere right now since our deprecation of
It's impossible to set such a restriction on people writing unit tests.
To be clear, the existing VSTest test adapter by definition has to be an in-process DLL, and it's VSTest that's responsible for resolving the adapter's dependencies (since in this case, VSTest is the executable). And the proposal for the new VSTest system is based on launching executables with known command line switches to set up their bi-directional RPC system. There is no dependency resolution problem to solve for the unit test projects in v3. The choice for v3 test projects to be an executable is the way you solve the dependency resolution problem: by letting the compiler & linker do the jobs they were designed to do. The dependency resolution problem is purely a v1 & v2 problem, and it needs to end now. To use the example from earlier: if the code in v3's executable stub (aka, the code in |
|
@remcomulder wrote:
Sorry for taking so long to answer this (I didn't realize I'd left it unanswered). As you might've noticed if you've been following the replies to @davidmatson, the problem is 100% down to dependency resolution. Of course any process could load another .exe file just as if it were a library, but that completely bypasses the dependency resolution that needs to be done which is what forced us down this path for v3 in the first place. If it's a solvable problem, it's not solvable by me in a reasonable amount of effort, and I've spent more hours that probably most people thinking about this problem (and trying to solve it once). It's just not reasonable. Even if someone provided a bulletproof PR to do it today, I'd still be skeptical about accepting it because I'd be the one maintaining it. How do I know it's bulletproof? I thought what I did (and kept patching) was bulletproof but edge cases cropped up immediately and plentifully. And then as time goes on, it would have to be evolved to support anything new that showed up in future versions of .NET. No, this is not a path I ever intend to go down again. I would sooner remove runner utility entirely and tell everybody to depend on a built-in VSTest adapter for 100% of execution (and I strongly dislike the way VSTest works, so you know this would be a "scorched earth" kind of scenario for me). |
|
Thanks, Brad. That really clarifies and makes sense to me. Regarding test exe dependencies: completely understood and agreed.
One problem that I don't think the VSTest adapters for C++ frameworks (at least GoogleTest & Catch2, if I recall correctly) really solved well is knowing whether a test exe is "theirs" or not. If I recall correctly, the Catch2 Test Adapter finds nothing by default, just to be safe, since it doesn't know if executing any exe it finds might do something very destructive (consider something in bin like ignore_any_command_line_args_and_format_all_drives.exe). I wonder if solving that problem well could go hand-in-hand with "negotiation" about how to invoke the exe (and builds on top of something xUnit already does, in seeing if xUnit is referenced). For example, could the in-process runner API have an attribute on it that indicates what version of the command line protocol it supports, and could the presence of this attribute be part of what the VSTest adapter for xUnit uses to decide whether to run the exe at all or not? Would that both give a measure of protection, in avoiding running any random exe, as well as allow the runner to use newer switches for newer exes? |
Yeah, this makes sense to me. Would you think then that the bi-directional RPC protocol is something that's implemented solely by the VSTest adapter for xUnit (v3)? |
There is a potential avenue available to us in .NET: add an assembly attribute dynamically at build time which indicated that this was an xUnit.net v3 test project, and what "version" of command line switches could be depended upon, something like: [assembly: XunitV3Project(protocolVersion: 42)]At the moment we look for DLL references to xunit/src/xunit.v3.runner.utility/Frameworks/XunitFrontController.cs Lines 177 to 189 in 27aad22 This is not particularly great at trying to prevent bad actors for launching things. For NativeAOT projects isn't going to work, since we'll only have a single stand-alone executable to look at, so the attribute based solution might be required, assuming that could be made to work with NativeAOT. |
There is no "test adapter" in their new model: the test executable is expected to be able to support their (as yet undocumented) launch command line switch(es) and do the RPC protocol. What they've done in their first effort to bring along existing VSTest-supporting unit test frameworks is to provide an executable stub (via a combination of NuGet + MSBuild) for an existing "old" VSTest adapter (like At the moment it's not clear to me that they have a plan which would just transparently support both, but they're still iterating on it. Being able to transparently support both would be critical for me to adopt the new system, because I don't see any particular benefit (other than performance) to supporting their new model. AFAIK, they don't intend to back-port this to older versions of Test Explorer, which is a big part of the blocker to me. So if you just "run" a v2 project with their proposed adapter, you'd get a command line UI that's provided by the VSTest team (and is super sparse at the moment). So you could do that today, if we were to merge their changes (and looking at their PR, you can see that I've made some changes and I'm able to run that locally). Here is an example from their sample:
The sample project is just a normal xUnit.net v2 project, but referencing the updated NuGet package the end result is something that's executable. It also uses a hack to intercept
Their answer to this is " For our v3, we wouldn't use their stub, nor would we require the user to add
|
I think they'll still need an adapter/bridge for some of the support that exists today; for example, for GoogleTest, Catch2, and Boost.Test. I doubt those frameworks will directly produce executables that speak this JSON-RPC protocol (and I think it's good that they be decoupled). I suspect such exes could still be supported with an adapter/bridge in between that knows how to turn the JSON-RPC protocol into calls to the test exe.
Yeah, I don't find that compelling either. Many, many existing test frameworks use console runners that output to stdout, and I don't think they were all wrong : ) I wonder if "dotnet test" is better replaced by "dotnet exec" for xUnit v3 though (maybe unless you want something more vanilla/framework-agnostic/MSTest/VSTest-like, such as TRX output).
Yeah, I wonder if the decoupling benefits exceed any perf improvement here. And I wonder if even better perf would be available with an xunit.runner.v3.visualstudio test adapter that runs the adapter itself in-proc with vstest.console.exe but the test out-of-proc in the xUnit v3 test exe. It's just that the communication between those two processes could be xUnit's output format, and not need the least-common-denominator impact of another layer of protocol between the test and the results in places like the VS UI. In other words, I wonder if there's both better decoupling and better perf in this combination:
than this one:
which, for existing VSTS frameworks that use exes, like the C++ ones, where they presumably wouldn't be adopting this RPC protocol anyway, would end up being:
Thoughts? |
For running an individual project, yes. That said, I could probably hack into MSBuild using the same technique they did, and make
I mean, that was (and still is, I guess) the plan when I get But it does come back to the debugger problem. I can solve that in Visual Studio, and I can't solve that anywhere else, because Microsoft does not open source the debugger and does not provide an APIs for me to be able to do a child process attach, which is why I'm actually excited for them to do this new design, even if it's not as good or flexible as whatever design I was planning. As to your comparison: either way, the stand-alone executable has to have some way to communicate with the meta-runner. I doubt choosing console would be "faster" than a protocol over TCP, because fundamentally you'd be getting messages as JSON in either way, so the creation/parsing expense is the big piece, and the underlying communication channel (console stream vs. TCP stream) will negligible.
I assume every test framework except for xUnit.net (and maybe MSTest) will be using their adapter against the old object model. If a test framework already implements against
For 1, are you assuming the third party test framework already produces an executable rather than a DLL? Which ones do? For 2, their model is that there is no such thing as a test adapter. It is required that the test project become an executable, and their bridge is what does that. The bridge, which I consider to be the degenerate form, leverages an old-style adapter DLL by providing the front end executable entry point while converting the test project from DLL to EXE. So in a sense there's an "adapter" here, but only when a test framework chooses to write to the old APIs (which are a plugin model) and not support the new APIs (which are a JSON-RPC model). That's why in my prototype there's no more adapter; it's already an executable, it doesn't care about the old object model, it just talks JSON-RPC directly. And I want to leverage that exact same mechanism for our meta-runners (which I suppose in theory would mean, if written correctly, our meta-runners could run any test project that observes the JSON-RPC protocol and not just xUnit.net v3 projects). For 3, prior to xUnit.net v3, I never had to mentally separate "in-process runners" vs. "meta-runners"; the latter is a term I coined for myself to describe the new differentiation between a test project that could run itself vs. a stand-alone executable whose job it is to run test projects in other assemblies. Mechanically, many (if not most) of the meta-runners are actually plugins into other environments (like Test Explorer/TestDriven.NET/Resharper plugged into Visual Studio, our MSBuild runner plugged into MSBuild, the VSTest runner plugged into VS Code, even probably architecturally the VSTest support built into Rider). Putting this all together, though, remember that every EXE layer you add introduces potential performance issues (since you'll have to communicate across that border), and more importantly it introduces the debugging problem. Child process debugging turtles all the way down. And adding executable layers here requires that the work be done by Microsoft, unless you're willing to give up on Microsoft's debugger. If I have to solve the child debugger problem myself, it probably means relying on an open source debugger like https://github.com/Samsung/netcoredbg. |
|
BTW, if you want to follow the thread where people are expressing their displeasure with the new extra-silent output from |
True, though if the format is something like gtest's URL-encoded event lines, that's quite cheap, and I think the protocol overhead would exceed the parsing cost. I guess it's a question then of the standalone EXE's output format. Mentioning perf here was mainly with this in mind:
i.e., if perf is the one possibly compelling feature of the new testfx runner, and if perf for xUnit v3 would be even better with in-proc testadapters with the current meta-runner, does sticking with it make more sense?
I think that's a safe assumption. It's actually called the MSTestRunner a number of places, it lives in MSTest's codebase, and it's used there as an opt-in feature.
This appears to be the case for every major C++ test framework I've seen; my first-hand experience has been specifically with GoogleTest, Catch2, and Boost.Test, which all have VSTest adapters to support running in the VS UI.
For any existing test framework that already uses exes, I think there are three options:
Agreed. The fact that the new MSTestRunner requires the target to be an EXE, and not all existing framework that produce EXEs are likely to support it directly, means one more process for those cases (or not supporting those frameworks, which means it might not have significant reach outside its home in MSTest). Supporting it directly in the test EXE also means bundling a JSON-RPC test server, which might make this kind of goal harder:
For xUnit v3, I'm guessing a VSTest adapter will be needed regardless (to support the VS UI, and back-compat with dotnet test/vstest.console.exe). I'm also hoping there's a simple way to get machine-readable output directly from an xUnit v3 exe, with full-fidelity xUnit output (rather than just least-common-denominator data). (Having a native output format is something that seems very standard in the C++ test world, and I've grown to appreciate the decoupling and full-fidelity advantages.) What do you think are the biggest pros/cons of v3 shipping VSTest integration (integrate via an adapter DLL) vs. also testfx meta-runner integration (bundling an implementation of the testfx JSON-RPC protocol inside each test EXE)? |
I'm still skeptical as well, though less so because I want to allow them to solve a real problem for me (debugging outside of VS).
Actually, what they're doing is replacing one process with another. Their current model uses external processes (that they control) into which things are loaded. Your tests don't ever get loaded into the process space of Visual Studio; instead, they launch an executable which communicates with Test Explorer via a proprietary communication channel. The design of this was at least partially in support of the VSTest's original support for more than just .NET Framework (that is, launching and running tests inside of a virtualized device for Windows Phone, or a virtualized sandbox for Windows 8). In fact, they have two separate executables processes: one for discovery (which stays alive) and one for execution (which only exists while tests are being run). This cross-process system is what necessitated us to add test case serialization, since we need to be able to "package up" the entire set of information necessary to run a test without being able to re-discover it. So in terms of executable count, the new system seems no worse than the existing system; they've just pushed the executable onto the test project itself instead of their own executable to host the adapter and test project.
I agree it's not ideal, though I think because the system is purpose built with limited interactions, it can be done without a full blown "JSON-RPC framework". The biggest cost here is that there is no HTTP server built into the BCL; instead, it currently lives in ASP.NET Core. All of this explains why my original plan was just a simple TCP-based solution, not HTTP (and yes, your proposed console solution is effectively the same thing, just on a different transport layer). Plus, I haven't seen any indication from the VSTest team that they ever plan to use HTTP; JSON-RPC is just the encoding protocol, and they liken it to the way the LSP is done in VS Code rather than HTTP-based JSON APIs. I don't know if it's too late to influence their design away from JSON-RPC or not, but it may be worth pushing them to reconsider it, especially in the face of their acceptance (seemingly, from that Expecto PR) that they consider it completely legitimate for test frameworks authors to re-implement the protocol themselves without using the Microsoft-provided bridge. That said, I'm not sure picking an alternative message encoding protocol is any better, just different.
That is where the
I'd say the advantage is "don't implement things twice", in the case of VSTest integration, if at all possible. I don't like their bridge implementation (for many of the same reasons as Expecto), and their bridge as it exists today doesn't actually solve my debugging problem without a lot of massaging; I can't end up with an executable calling an executable, it has to be the case that enabling VSTest support modifies the existing test project, like it does in v2, except that the integration for v3 will by definition have to look much different. I don't know what that looks like yet, given that my initial prototype on the v3 side has looked for ways to directly support it. I had originally assumed that an iteration of This appears to be the extent of the required dependencies to bring in both the bridge and the MSBuild hack to make
So nothing concerning for the general xUnit.net user. Everything else comes from the BCL. Note, though, that this does not yet contain (to the best of my knowledge) any JSON-RPC client other than the named pipe client; they've suggested that Test Explorer integration will not use named pipes. So this may change slightly for Test Explorer, but since they only have to implement the client side things, they should be able to just use |
|
Thanks for the detailed and helpful discussion here, Brad.
Yeah, though it feels a little dirty, avoiding a full blown server framework could be feasible. For a somewhat related example, GoogleTest's support of JSON and XML output is implemented via plan old string concatenation (plus an encoding/escaping function, for user-provided data). A similar, "roll-your-own" JSON-RPC server could work, though the two-way nature of the protocol might make it notably more challenging than dependency-free writing of JSON or XML.
Yeah, I'm just stealing this idea from C++ test frameworks - I've had to integrate three of them with other test infrastructure, and I found their pattern of "command line for input, stdout (or maybe files or write-only TCP) for output" to be suprisingly simple, complete, and easy to use/invoke. When I heard xUnit was moving to exes, I thought it was worth passing along the idea of using the same pattern I'd enjoyed so much with C++ test exes; I've been really impressed by its elegance.
I think that's how the adapters for .NET test frameworks work today, but I wonder if it isn't actually a core part of the current VSTest contract. I haven't checked, but I wonder if it's one worse for some of the existing exe-based frameworks - for example, I wonder if the Boost.Test adapter today goes through the testhost.exe layer, or if it directly invokes the Boost.Test exe (I'd guess direct invocation, since it's not a .NET exe anyway to begin with). Assuming there's a way to launch test exes directly from an in-proc DLL in VSTest today, that might be a net regression?
If I had to guess, I'd suspect they'd be fairly highly motivated. I think MSTest is the only current consumer of this runner (since they're the ones who built it). If they could land xUnit buy-in for their new solution, I think that would put them at/over the tipping point in terms of critical mass for .NET, so I think you'd be in a strong negotiating position : )
The main possibilities of "better" I would see are:
Yeah, this makes sense. Having a one-way output stream makes it much easier for to integrate with other testing infrastructure - it avoids the need to select, get access to, and consume support code for a more complex protocol, which can be difficult when consuming from a different language that doesn't have as extensive and simple library support. Vanilla text lines over TCP aren't too bad either. JSON, for example, is a little harder that QueryString decoding (requires a JSON parser rather than just a query string decode function), but not enough to be a deal-breaker. JSON-RPC might be hard enough to give up, if runner/results integration code is written in a language like C++. The top C++ test frameworks all doing output-only streams, and how well that seems to be working, with its simplicity and decoupling, weighs in favor of that approach for me (prior art). I'm mainly thinking here of the "contract" of a test EXE. What are its fundamental capabilities? How are they invoked? How is output provided to a human or to a machine? Having the command line and console output being so easy and natural to use (when it's already a console exe) seems hard to pass up.
Yeah, having a solution to the debugging problem would be really nice. Maybe that's something VSTest could add support for, and then the "don't implement things twice" option would be only ship the VSTest runner, but bypass testhost.exe and launch an xUnit v3 exe directly? (If VSTest supported that, would testfx add anything?)
I tend to see support for dotnet test/CI and the VS UI as the main reasons to support a meta-runner, which VSTest already has (and, perhaps, with fewer dependencies required to be carried along inside the test exe). Maybe it's worth waiting to see what the cost is to get Test Explorer integration via the testfx runner compared the VSTest runner? |
I'm unaware of any way to opt out of the separate executable in today's VSTest model. That doesn't mean it doesn't exist, but I'm not aware of how it's done. If that exists today, then that would theoretically solve my debug problem, since they must have the same debug problem. It would be interesting to verify this hypothesis and determine how it's done.
I'm of two minds here. I assume they're trying to get something shipped in the next drop of Test Explorer in VS. Being unable to update it out of band is definitely a limiting factor for them. I absolutely hate installing pre-release versions of VS and .NET, but I could put something into a VM and try to see where they are in terms of support there. I'm worried that once they ship something, a lot of stuff goes from liquid to concrete, so the time to realistically influence may already be past. |
You're right; I had a GoogleTest project handy, and I just checked Debug via Test Explorer, and it does have a parent testhost.exe process - the process tree is vsts.console.exe -> testhost.exe -> some_googletest_exe Having a way to skip the testhost.exe layer here would be really nice, and likely would address the debugging part as well (which probably applies to these other frameworks as well). |
|
Did VSTS add a solution for the debugging part recently here? |
|
Wow, that's a great find. I can verify that it's in 17.9.0 at the very least.
I'll need to do a little experimentation to ensure it's working in VS Code. I strongly suspect that it won't be available in third party runners like Rider, but that would also be a good experiment to conduct. |
|
Cool - glad that helps! It's also possible there's a way to bypass TestHost.exe, using a .RuntimeProvider.dll. Apparently Python and TAEF (largely an internal Microsoft test framework) had some need way to do it, from the RFC above. I haven't had a chance to confirm yet though or see exactly what that would extension point would enable. |
|
I don't know if bypassing the testhost helps us, since we have to start with the adapter inspecting the assembly to determine if it's v1/v2 (in which case it's hosted in-process, but maybe in a separate app domain in some cases) or v3 (in which it's hosted out of process). Providing two different adapters is a possibility, I suppose, but that seems like an optimization for down the road, and then it becomes a weird interplay problem (they'd have to be separate NuGet packages, since obviously NuGet would have no way to differentiate between a reference coming from xUnit.net v1/v2 vs. v3). |
|
Hmm, interesting point. What if the runtimeprovider checked for v3 vs earlier in-proc first, and then delegated to the normal runtimeprovider if it found a pre-v3 DLL? (Or maybe it could even just check the file extension, so it wouldn't even need to load the assembly?) |
|
Things to consider for sure. |
|
So, to jump back to a console vs. TCP related part of the discussion: How would you support cancellation? One of the things we currently support is the notion that you can request cancellation which allows the tests that are already running to finish without hard aborting them. The way this is surfaced in our console runners (both in-proc and meta-runner) is "Press Ctrl+C once, we'll ask for cancellation; press Ctrl+C a second time and we'll just shut everything down". For v1/v2 (and in-process) we use APIs to request that cancellation. With v3 being a separate process, would we just keep the Ctrl+C handling as-is? I wonder if there's a simple x-plat way to do that. Just send |
|
Good question. Ctrl+C for graceful cancellation vs Ctrl+Break for forceful abort? I think I've seen some container stuff do that. If I recall correctly, it corresponds to SIGINT vs SIGTERM on *nix (or maybe vice-versa). Edit: (I'm not sure whether they have a nice cross-plat way to invoke Ctrl+C/sigint. Worst case, I think it would be P/Invoke of GenerateConsoleCtrlEvent on Windows and I'm not sure what on *nix) |
|
Also, just keeping the current "send Ctrl+C once/twice" seems a very reasonable option. And I think having a stdin char for that could also be really simple. It's not quite as "standard" as the official Ctrl+C signal, but I like the simplicity, and as long as Ctrl+C still works, I don't see much downside. |
I didn't do that for interactive usage because honestly there are a decent number of keyboards out there that don't even have a Break key any more, especially on laptops. I'm not 100% sure that .NET can differentiate those anyway (I guess if Ctrl+Break just kills the process unilaterally then there's nothing to intercept 😂). |
|
Yeah, that's correct; Ctrl+Break is really forceful and can't be interecepted. If you want a more insistent graceful shutdown, Ctrl+C twice is probably the best option. |
|
Whether I use this for the v3 RPC or not, it seemed like an obviously good idea to add an In the process of trying to reduce & remove dependencies, I noticed I had a JSON obviously doesn't have any notion of types, so I threw a magical Examples running a single test (piped through jq just for readability, the norm is one object per line):
{
"Type": "discovery-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"AssemblyName": "xunit.v3.assert.tests, Version=0.1.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c",
"AssemblyPath": "C:\\Dev\\xunit\\xunit\\src\\xunit.v3.assert.tests\\bin\\Debug\\net472\\xunit.v3.assert.tests.exe"
}
{
"Type": "discovery-complete",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCasesToRun": 1
}
{
"Type": "test-assembly-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"AssemblyName": "xunit.v3.assert.tests, Version=0.1.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c",
"AssemblyPath": "C:\\Dev\\xunit\\xunit\\src\\xunit.v3.assert.tests\\bin\\Debug\\net472\\xunit.v3.assert.tests.exe",
"Seed": 1114100141,
"StartTime": "2024-04-16T16:33:16.6089932-07:00",
"TargetFramework": ".NETFramework,Version=v4.7.2",
"TestEnvironment": "64-bit .NET Framework 4.8.9232.0 [collection-per-class, parallel (24 threads)]",
"TestFrameworkDisplayName": "xUnit.net v3 0.1.1-pre.402-dev+7864e5e082"
}
{
"Type": "test-collection-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestCollectionDisplayName": "Test collection for BooleanAssertsTests+False (id: cb82e8ecacbab286cb49cdb09257c46444784190d9768a2ed3feb31f55021add)"
}
{
"Type": "test-class-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestClass": "BooleanAssertsTests+False"
}
{
"Type": "test-method-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestMethod": "AssertFalse"
}
{
"Type": "test-case-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestCaseUniqueID": "e01e0b7d2048d8b1983a6404c17af5b6a7016c502c03fe24ff979c153bd6ad5a",
"TestCaseDisplayName": "BooleanAssertsTests+False.AssertFalse",
"Traits": {}
}
{
"Type": "test-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestCaseUniqueID": "e01e0b7d2048d8b1983a6404c17af5b6a7016c502c03fe24ff979c153bd6ad5a",
"TestUniqueID": "2f44cccb34ca57ff3352278fa6a541cfc7655ef44cb3672ceb1f69e6b79fa9d1",
"Explicit": false,
"TestDisplayName": "BooleanAssertsTests+False.AssertFalse",
"Timeout": 0,
"Traits": {}
}
{
"Type": "test-passed",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestCaseUniqueID": "e01e0b7d2048d8b1983a6404c17af5b6a7016c502c03fe24ff979c153bd6ad5a",
"TestUniqueID": "2f44cccb34ca57ff3352278fa6a541cfc7655ef44cb3672ceb1f69e6b79fa9d1",
"ExecutionTime": 0.0051102,
"Output": ""
}
{
"Type": "test-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestCaseUniqueID": "e01e0b7d2048d8b1983a6404c17af5b6a7016c502c03fe24ff979c153bd6ad5a",
"TestUniqueID": "2f44cccb34ca57ff3352278fa6a541cfc7655ef44cb3672ceb1f69e6b79fa9d1",
"ExecutionTime": 0.0051102,
"Output": ""
}
{
"Type": "test-case-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"TestCaseUniqueID": "e01e0b7d2048d8b1983a6404c17af5b6a7016c502c03fe24ff979c153bd6ad5a",
"ExecutionTime": 0.0051102,
"TestsFailed": 0,
"TestsNotRun": 0,
"TestsSkipped": 0,
"TestsTotal": 1
}
{
"Type": "test-method-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"TestMethodUniqueID": "cbcb8736752996febb7f8d31d72cdd267a31db8651edc14a99431fce6724d6a7",
"ExecutionTime": 0.0051102,
"TestsFailed": 0,
"TestsNotRun": 0,
"TestsSkipped": 0,
"TestsTotal": 1
}
{
"Type": "test-class-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"TestClassUniqueID": "f8f0c278fc4fa831e7ebb141810bd1aaed0f3e1a57815824d06fa3cd3acac03b",
"ExecutionTime": 0.0051102,
"TestsFailed": 0,
"TestsNotRun": 0,
"TestsSkipped": 0,
"TestsTotal": 1
}
{
"Type": "test-collection-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"TestCollectionUniqueID": "bd32505122966234e4755fe8cc0c189c6939ec2336027b433178d6dd901a4172",
"ExecutionTime": 0.0051102,
"TestsFailed": 0,
"TestsNotRun": 0,
"TestsSkipped": 0,
"TestsTotal": 1
}
{
"Type": "test-assembly-finished",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"ExecutionTime": 0.0690445,
"TestsFailed": 0,
"TestsNotRun": 0,
"TestsSkipped": 0,
"TestsTotal": 1,
"FinishTime": "2024-04-16T16:33:16.6825080-07:00"
}And some simple error handling:
{
"Type": "diagnostic",
"Message": "error: unknown option: -foo"
} |
|
This looks very nice - thanks, Brad! |
|
Hi. YoloDev.Expecto.TestSdk maintainer here. I figured I'd chime in and also ask some questions as I believe we are in a somewhat similar situation and it seems like you've looked a bit further into this new test platform than I have. I currently have no plans of accepting the So I started thinking about what it would take to support both new and old test explorer and So while this issue is mostly about the meta-runner side of things, I wanted to show my interest in how you're dealing with the test-embedded-executor side of things. Cause while I'm definitely one for re-inventing the wheel - the current implementation of YoloDev.Expecto.TestSdk has a lot of learnings taken from the xunit implementation :). Also, I really think a better and more defined glossary might be useful - cause there are a couple of pieces of my mental model that I don't yet know are correct or how to name. So I'll try to explain my understanding of the pieces, and you can correct me where I'm wrong, or our understanding differs: meta-runnerThese are what I would typically call a test runner/executor - but with the new model where test assemblies can run themselves, it makes sense that these need a new name. I personally think the name "meta-runner" is a bit unfortunate and would probably rather call it a test-orchestrator (or similar), but regardless it's an executable/library that's responsible for discovering testable assemblies/programs and then invoking/loading them for discovering/running tests. In the v2 model, these runners would load an adapter for the tests, look for an implementation of In the v3 model, these runners discovers test-executables somehow, and then simply call these executables with command line flags and sets up some sort of RPC channel to interact with the test execution. adapterThese are dlls that are in the v2 model loaded into the meta-runner and provide implementations of self-execution-stubTime for me to coin my own term I guess (please help me come up with a better name for this). The v3 model requires test assemblies to be executable, which means they require a It's responsible for implementing the process-contract for v3 testing - ie. reading command line arguments, setting up any required IPC, discovering and running tests etc. I'm on very thin ice for this one, and my understanding is fairly rudimentary. I don't know what kind of This assembly can also probably be an adapter to provide backwards compatibility for v2 meta-runners. |
Sorry, I think I've done an appalling job of explaining this issue from my side. To use the terminology described here, NCrunch (my runner), fits the description of a meta-runner. As such, we orchestrate the test run. We distribute it over multiple machines, report test results, etc. To make this work, we have about 15 years of R&D in handling exactly the assembly resolution problems you've described. You are right to want nothing to do with these problems, because they are horrible. If I understand correctly, you want to delegate the assembly resolution out of xunit so that this is handled instead by MSBuild and the platform itself, by making the test project executable. This sits fine with me and I think that we can work with this. Something we can't do is lean on the same MSBuild system to handle that assembly resolution. We need to use our own, which means that we need our own code to be responsible for building the test process, handling the RPC, etc. So what I need is a way to work with xunit in-memory. Calling the main() method in the test assembly (from my own test process) will probably get me almost all of the way there. However, if I'm doing this, then I'm by-passing xunit's API and simply calling its types like a service. It also means that I need to build an RPC/IPC pipe that is compatible with xunit's results reporting, so we can get the data from the test run. Again, this means I'm building under the hood of xunit .. which doesn't feel like the goal here. What would be the ideal solution to this? Could xunit's IPC system be modular enough that I can use xunit's meta-runner side of it as a service? Or is there a way for xunit v3 to simply let me handle all the dependency resolution and process sandbox so that it doesn't need to? To be clear, I'm not trying to tear down your chosen path with this design. I've spent 15 years of my life in dotnet dependency hell and I'm glad you've found a way to escape from it. All I need is a hook somewhere that will let me by-pass xunit's call to Process.Start. You would be responsible for none of the dependency hell or the consequences of its resulting insanity. |
|
@Alxandr wrote:
Yeah, I don't love the name "meta-runner", but the alternatives all seem to involves a lot more syllables. 😂 "Multi-assembly test runner" is probably a better name, to differentiate it from a project which is self-running (and can only run itself, not anything else). Such multi-assembly test runners are responsible for orchestrating parallelism across assemblies when possible (it's out of our control in the case of VSTest, unfortunately).
Complications with App Domains aside, this is close enough to correct.
Remembering of course that our "runner utility" system allows for backward compatibility, such multi-assembly test runners need to be able to run any of v1, v2, or v3 projects. As you point out, when running v3 projects, they would launch the test project as a separate process (as they require) and communicate via some RPC channel. The channel is yet to be defined, though at the moment I am tending towards a model with mostly one-way communication: launch the executable with known command line switches (to match what the user has requested; for example, to filter tests by class name), and then use the console to report back the messages (encoded as JSON) that would've normally been exchanged as objects in v2 (and XML in v1). I really have ended up with a whole mess of protocols. 😄
Custom test frameworks (that is, starting with something that implements
We ship auto-generated entry points for C#, VB, and F#: https://github.com/xunit/xunit/blob/main/src/xunit.v3.core/Package/content/AutoGeneratedEntryPoint.cs They can be disabled with this following in your project file: <PropertyGroup>
<XunitAutoGeneratedEntryPoint>false</XunitAutoGeneratedEntryPoint>
</PropertyGroup>In which case you're responsible for creating your own Main method. The implementation of the auto-generated version is straight forward: public class AutoGeneratedEntryPoint
{
public static async global::System.Threading.Tasks.Task<int> Main(string[] args)
{
return await global::Xunit.Runner.InProc.SystemConsole.ConsoleRunner.Run(args);
}
}(Note: this is an implementation detail that may still evolve before v3 is finalized.)
Yes, although we provide classes to do all of this. One of the pieces of feedback we'd want is how split this process of command line parsing from execution is; today, they're both bundled up into
At the moment, we allow the override of Main because we don't know exactly what or how people might want to set things up before running their tests. For example, you could imagine that one way to test a service would be to get the service running before executing any of the tests, and then cleaning up the service when the tests are finished. This might include fully supporting the real startup and cleanup of the service (like setting up a DI container). ASP.NET Core has done some clever workarounds here to work in the confines of what's available today, but I wanted to enable more powerful and flexible options. That said, I haven't released v3 yet, and as far as I know, I'm pretty much the only one using it right now, so I don't really have any idea how people are going to want to use this system. I expect there to be a significant runway of pre-release usage, especially from developers who currently write third party extensions, to try out what's available in v3 and provide feedback on what works and what doesn't. |
|
@remcomulder wrote:
This is correct (technically we're letting the compiler do the work, but that's nit-picking).
Just to verify: you're using xunit.runner.utility today to discover & run tests, right? Whatever changes we'd need to make to the xunit.v3.runner.utility to enable this "load in process" model would feel like the right place to insert the required changes.
I agree that being that low level isn't ideal. We haven't hidden those details in the past, but we also haven't supported the idea that users would ever play at that level with any kind of support from us. To the best of my knowledge, nobody does... or if they do, they've managed to do it without ever asking me a single question about it, which would be impressive. Whatever the RPC model is for v3, we would never anticipate someone intercept or interact with it directly. All those details would be hidden behind runner utility code; in the case of adding "load in process" support for v3 this would mean that we would call Main with the arguments on your behalf, rather than "invoking" Main by launching the process with the correct arguments. I haven't really worked through where the hidden gotchas might be here, but I'd absolutely be willing to work with you to ensure you could get access to the in-process loading feature with minimal effort (to the point where presumably you shouldn't even care whether it's a v2 or v3 project, either way the runner utility abstractions would hide that from you). To ensure that any plans we make would work, I think I'd want to know more about how you handle dependency resolution today, but that's probably a bigger discussion than we should do in a GitHub issue. I'd be happy to do a call so you can screen share and you can show me enough for me to be frightened but knowledgeable. 🤣 |
|
It's also probably worth mentioning that the JSON objects you saw in the examples I posted above are basically the same object model that I would anticipate would be available to people in xunit.v3.runner.utility. Those messages today look like an improved version of what was available in v2 and defined in Xunit.Abstractions. They live here today for v3: https://github.com/xunit/xunit/tree/main/src/xunit.v3.common/v3/Messages Compare this: {
"Type": "test-assembly-starting",
"AssemblyUniqueID": "a9a829c1e48913d5b81435c80ce97b5634bb5df1be3bd7bccb66b85cdaf33af3",
"AssemblyName": "xunit.v3.assert.tests, Version=0.1.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c",
"AssemblyPath": "C:\\Dev\\xunit\\xunit\\src\\xunit.v3.assert.tests\\bin\\Debug\\net472\\xunit.v3.assert.tests.exe",
"Seed": 1114100141,
"StartTime": "2024-04-16T16:33:16.6089932-07:00",
"TargetFramework": ".NETFramework,Version=v4.7.2",
"TestEnvironment": "64-bit .NET Framework 4.8.9232.0 [collection-per-class, parallel (24 threads)]",
"TestFrameworkDisplayName": "xUnit.net v3 0.1.1-pre.402-dev+7864e5e082"
}to this: https://github.com/xunit/xunit/blob/main/src/xunit.v3.common/v3/Messages/_TestAssemblyStarting.cs |
Yes, right now we use only xunit.runner.utility. It works well for us and we've had minimal issues over the last few years.
This would work fine for us. At the time we start interacting with xunit.runner.utility, we're already in a carefully constructed sandbox process designed specifically for test execution, with all the dependency issues handled. A consideration for a 'load in process' setting would be that without further modification, this would probably cause a loopback IPC connection inside xunit, since I guess the rest of the system would expect to be pushing data through a pipe of some kind. Probably this won't be as efficient as just using in-memory, but I'm happy to accept it if it keeps things simpler for you.
I'm happy to help in any way that we can make this happen. I really want to use your API. I think it's better for everyone this way.
I think a call would be a great idea :) I'll PM you and we'll see if we can set something up. |
|
I might not have been clear enough in my original post. I'm not trying to integrate with xunit itself, I'm (same as xunit) wanting to integrate with the new microsoft test platform without having to depend on non-FOSS libraries. Basically, I want to do the same as xunit on the test library side and the
Or just test-orchestrator? xD
Apologies, but I was probably a bit unclear here. I'm talking about
This is very interesting. This means that you could in theory code-gen/embed the entire RPC implementation into the test assembly (though that's probably not a good idea). |
|
@Alxandr wrote:
Gotcha. Now, this is all assuming you're trying to implement the new RPC mechanism, and not the old ObjectModel system. Obviously implementing ObjectModel doesn't require any closed source anything, and you can just look at our source in https://github.com/xunit/visualstudio.xunit for an example. As for the new RPC mechanism...the only way I've gotten anywhere in my own personal implementation is by reverse engineering (meaning, ILSpy) the closed-source Microsoft implementation of the adapter. I have a branch for this against v3 (https://github.com/xunit/xunit/tree/microsoft-testing-platform) which is now a few months old, because it was mostly just a proof of concept. It utilizes And to be clear, none of this involves anything in ObjectModel, because that's the old API. Instead, this uses their
This was my intention, to have all v3 test projects natively support the new VSTest system without any need for adapters, since the logic all seems to be around parsing special command lines and handing off to their RPC implementation. Doing this via an adapter package of some kind would add layers of complexity that don't seem like it would be worth it, though I'm not averse to hearing architectural suggestions. The amount of code necessary to integrate this into our in-process runner is fairly minimal, though I'm not wild about the required There is the further issue that, right now at least, you still need the adapter for the older API so as to continue to support older versions of Visual Studio, as well as third party runners (like Resharper/Rider) that are consuming unit test projects via the older API (they effectively replace Test Explorer, so they're on the other side of the RPC mechanism). It doesn't seem like there was any specific effort added to ensuring backward compatiblity, and in fact the VSTest team's eggs all appear to be placed into the basket of "let the unit test frameworks continue to implement the old ObjectModel API and we'll adapt to the new system for them" which is in stark contrast with v3's executable architecture, since they want to become the entry point (and convert your test project from DLL to EXE). I guess I'll know more for sure once I get xunit.v3.runner.utility which can run v3 projects, and then see how an updated version of the Visual Studio adapter works in the face of the old APIs. I have asked for things to be documented (and opened up) and the team has been very slow to answer and with not much detail yet. I opened several issues, most of which have been closed, though there is this one still open relating to documenting the IPC mechanism: microsoft/testfx#2539 Almost everything they've said so far has amounted to either "we don't plan to do/document this" or "we haven't documented this yet". Until they document things officially and make it easy for me to integrate with their implementation (or document it fully enough so I could create my own RPC implementation), then my work here remains just a proof of concept that I honestly don't spend any time thinking about. If you're headed down this path...good luck. Let us know if you make any headway. |
This is my stance as well. edit |
Given the v3 architecture where unit test projects are programs means we need to add support in v3 runner utility to be able to launch these separate processes for meta-runners (like
xunit.v3.runner.console).More details TBD.
The text was updated successfully, but these errors were encountered: