Aspire.NET support for xUnit v3

[joshmouch]

I'm not sure the best way to cross post, but I'm basically trying to find out if xUnit v3 is/can/will be supported as an Aspire.NET resource.

Here's my post: dotnet/aspire#3125

And here's the content from that post:

I'm trying to find the best way to add a test project (or DLL or EXE) to be run from an Aspire AppHost. When using Visual Studio and debugging with F5, Visual Studio should also attach to the tests (and test project).

If I were using xUnit v3 (which has an EXE entry point instead of dll), I think I could just use .AddProject("ExampleXunit3TestProject.csproj"). As far as I know, any environment variables defined by .WithReference() would remain intact while the tests are running. I don't know that for sure, though, since xUnit v3 generates a Main() that, in turn, creates an XunitConsoleRunner. That sounds like a new process to me (and therefore not attached to debugger), but still investigating. The author stated that VSTestRunner isn't available yet, which is the one that I think would attach the debugger to the new processes. @bradwilson

Using xUnit v2, you have to use a test runner since there's no EXE to call. xUnit has their own test runners (e.g. xunit.runner.console), but it'd be preferrable to use a test framework agnostic test runner, which means vstest.exe, or dotnet test or similar. In my initial testing, I lose my environment variables with dotnet test, which means the tests don't have access to the modifications to environment from .WithReference(). In other words, using .AddExecutable() with dotnet test results in tests not having access to .WithReference().

In both of the above, it's unclear to me if VisualStudio would attach to the processes being spawned. I guess that's up to how .AddExecutable() is implemented. If it doesn't today, then it'd be great to do so or else add an option to do so.

Ideally, it'd be great to add a new DistributedApplicationBuilder extension .AddDotNetCliCommand() or .AddTestProject() specifically to add test projects with the outcome of running them with dotnet test.

I think this also paves the way to being able to run tests from Visual Studio (or Resharper) UI into a docker container. To my knowledge, this is either not possible today, or not documented. This is assuming Aspire is going to allow F5 to put projects into a container instead of in process (through a launchProfile, perhaps?), and that it'll automatically attach to the process in the container. That or else use the trick that VisualStudio does with docker files where it mounts the EXE as a volume and just attaches to it locally.

[bradwilson]

Alpha docs are here: https://xunit.net/docs/v3-alpha

You can disable the automatically generated entry point:

<PropertyGroup>
  <XunitAutoGeneratedEntryPoint>false</XunitAutoGeneratedEntryPoint>
</PropertyGroup>

Now it's up to you to provide the entry point. The docs on overriding the entry point show what the current auto-generated entry point looks like. That said, if ConsoleRunner isn't doing what you want, you're not actually required to use it (this would be especially true if what you're creating isn't a console application 😂).

The in-process ConsoleRunner class does two major things: parse the command line options, and then use the xunit.v3.runner.common functions to execute the actions that the user has requested. Looking through the in-process console runner project shows that it's pretty shallow, so you should be able just pick and choose the bits the most sense for you. If there are non-public things that you'd like to have made public and/or APIs that could use review, open an issue and I'll take a look. This code has not gone through an extensibility pass yet (since we're not even out of alpha).

---

As for the question of attaching child processes to the debugger, there's a hacky way to do so when using Visual Studio, and no way to do so that I'm aware of with any other IDE (including VS Code). It would be much better for you to consume the unit tests as a library directly from your entry point rather than leaving it as an executable, and then letting the system launch you and you decide when/how to run unit tests. Even the fact that we report things to the console by default isn't necessarily a requirement; that's just how we choose to handle it by default.

I haven't tried this "leave it as a library which you import from an actual runner project" as an option yet, and there may be blockers in the current NuGet packages (since I'm pretty I try hard to emit <OutputType>Exe</OutputType>. If you experiment here and there are some changes that need to be done here, I'm happy do them to enable this scenario.

[bradwilson]

To be clear, I envisioned the idea that you'd want to set up an environment ahead of time (like a full ASP.NET app) before starting the tests, which is why the entry point is overridable. This sounds like exactly the kind of scenario that we wanted to enable.

[joshmouch]

I've got some work ahead of me. This is an awesome response, thanks!