Test explorer tree view groups [Theory] attributed tests differently
kahveci opened this issue · comments
I have a sample unit testing project with one test class including following unit test methods; two [Fact] attributed, another two [Fact] attributed but also Skip enabled, and finally, two [Theory] attributed with both having two [InlineData].
As you can see below, the [Theory] attributed test methods are grouped differently when you run tests.
On the other hand, this is not the case when you discover tests using refresh button. I reckon, all test methods of the same test class should be grouped together regardless of [Fact] and [Theory] attributes.
P.S. This bug report also has a relation with this one #200, but addresses a different aspect the current UX. For that reason, I thought it is better to report separately.
My system information:
Version: 1.33.0 (system setup)
Electron: 3.1.6
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17134
.NET Core Test Explorer: 0.6.6
.NET Core: 2.2
xUnit.net: 2.4
Thanks for reportingI think this issue is similar to the one reported in #159. As you can see there xunit and dotnet test have some inconsistencies when reporting the test results. I would also love that the tree behaves in an expected maner but trying to fix these type of "bugs" in our extension is really time consuming I'm afraid. But we are aware of it.
Thanks for your reply @stefanforsberg. I totally understand the situation. I will have a look at the code base and see if I can create a PR for this and related issues.
My workaround was to use [Theory][InlineData()] instead of [Fact]
I'll take a look as well.
@kahveci This issue continues to confuse me, just when I think I have a grasp on it I realize I don't 🛑
I have a test class which contains both Fact and Theory and they are grouped up in an expected fashion
What I do know is that if we have multiple namespaces in the same assembly (eg some tests are under NamespaceOne and others are under AnotherNamespace) things start to get iffy.
By the looks of your image you have everything under Playgroundx... is that so?
@stefanforsberg after your message, I tried a couple of different scenarios and managed to find the root cause I guess. Simply, your test class namespace must match with the assembly name. My assembly name is Playgroundx.LogAn.Tests while the namespace is Playgroundx.LogAn. I changed the test class namespace as Playgroundx.LogAn.Tests, and the problem solved as follows. Hope this helps fixing the issue.
@kahveci I've asked a question on the xunit slack and we'll see if that give any more information.
Xunit people claims the error probably lies within the microsoft vstest (which is the thing used to generate the trx-report files which we deduct the test results from).
If you want I think you should open an issue on the vstest repo and see if that gets any traction. It's an error that's very hard to try and solve in our extensions I'm afraid.
Many thanks @stefanforsberg. Would you attach the discussion with Xunit people if possible? It might be useful to talk to MS vstest repo.
Stefan Forsberg [11:06 AM]
Hello! I'm seeing some inconsistencies (at least I think they are) in how xunit test results are reported in a trx file created by the dotnet test command. Is this something that xunit has any control over or where does an issue of this type belong?
Basically Fact and Theory tests have different information in the className field.
bradwilson [5:01 PM]
The TRX file is entirely generated by VSTest, so likely nothing we can control causing your issues.
Stefan Forsberg [5:32 PM]
Ok, I'll try being more concrete and it might be easier for and decide the likeliness.
Without setting an assembly name in my tests csproj file I get the following results (abbreviated) :
className="XunitTests" name="XunitTests.TestClass2.SkippedTest"
className="XunitTests.TestClass3" name="XunitTests.TestClass3.Pass(value: 3)" />
When I set an MyProjectName in the csproj the results are reported as follows
className="MyProjectName" name="XunitTests.TestClass2.SkippedTest" />
className="XunitTests.TestClass3" name="XunitTests.TestClass3.Pass(value: 3)" />
Eg the classname seems to contain the assembly name but only do so for the Fact test and not the Theory test. Does this help?
bradwilson [6:22 PM]
Sorry, I don't think that's anything we control.
@stefanforsberg after your message, I tried a couple of different scenarios and managed to find the root cause I guess. Simply, your test class namespace must match with the assembly name. My assembly name is
Playgroundx.LogAn.Testswhile the namespace isPlaygroundx.LogAn. I changed the test class namespace asPlaygroundx.LogAn.Tests, and the problem solved as follows. Hope this helps fixing the issue.
This fixes my problem, thanks!
I've added a known issue for this and will close this in the mean time. Thanks a lot for your help in fleshing this out @kahveci and @MatthewS2077