How to run XUnit tests in parallel with scope? #2924
-
Question ❓I'm planning to use XUnit for integration tests, where only tests within the same namespace are executed in parallel and the other namespaces sequentially wait for the previous namespace to finish their run, before running their tests in parallel and so forth. Description:Here is how my project will be structured.
ℹ NUnit Example demoI've managed to achieve this functionality using NUnit, Here is the repo with the NUnit example. Here are the logs produced when you run all the tests. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Our parallelization right now is built on the concept of "test collections": Tests within the same collection do not run in parallel against each other, but tests from different collections can run in parallel. What you're asking for is basically an inversion of how our parallelism logic currently behaves (you want to group things for parallelism rather than sequentiality). More complicated here is that you're also asking for what is effectively a "collection fixture" (that's the Setup/Teardown you're doing at the group level), which in our model is further indication of sequentiality (because it's intended to share a piece of state across all the tests). Could it be done via extensibility? Almost surely. Would it be easy? Definitely not easy to figure out for someone who doesn't know our extensibility system well, which means mostly... just me. 😭😂 Let me see how difficult this would be, and if it's easily doable, I'll set it up as a sample project that you could then steal the infrastructure pieces from. It's gonna take a little time. |
Beta Was this translation helpful? Give feedback.
-
|
I have a sample that I believe does what you want. Attached below is the output I'm seeing. I have applied no sort order to the namespaces, so they're running in the order that they were discovered via reflection (or perhaps the order that LINQ chose to group them in), which is C => B => A. (I commented out all the I had to override a bunch of the execution pipeline so that things got flipped from sequential to parallel and parallel to sequential. The sample does not respect any of the fixture interfaces nor test collection definitions, but instead creates a test collection per namespace, and then uses that to ensure that we run the namespaces sequentially. Everything else (Test Class => Test Method => Test Case) becomes parallelized. Although your example didn't show it, I also provided parallel support for individual theory data rows (and one sample class that has that, Obviously, I tested this with your happy path (included in the sample) but have not stressed it to ensure that it fails in the right ways when necessary (aka, no unit tests). Source: https://github.com/xunit/samples.xunit/tree/main/NamespaceParallelization |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Brad! 🙌
Almost everyone mentions how extensible XUnit is but looking at the example you've setup, nope not easy at all 😅. |
Beta Was this translation helpful? Give feedback.
-
|
This is a category of extension that's basically like saying "I want to make my own test framework". We're defining a whole new set of rules and behaviors about how things are supposed to work, and the extensibility does allow for that, but makes you work for it a little bit. 😄 Most extensibility is not this level of rule-rewriting. |
Beta Was this translation helpful? Give feedback.
I have a sample that I believe does what you want. Attached below is the output I'm seeing. I have applied no sort order to the namespaces, so they're running in the order that they were discovered via reflection (or perhaps the order that LINQ chose to group them in), which is C => B => A. (I commented out all the
Assert.Fail()calls because they were just adding noise to the output.)I had to override a bunch of the execution pipeline so that things got flipped from sequential to parallel and parallel to sequential. The sample does not respect any of the fixture interfaces nor test collection definitions, but instead creates a test collection per namespace, and then uses that to ensure …