Out of order test method execution
Alastairm opened this issue · comments
When I have >12 tests in a class a test will execute its methods out of order.
Output:
Test 10
test method 7
test method 1
test method 2
test method 3
test method 4
test method 5
test method 6
Code:
public class LofuOutOfOrder: LoFuTest, IDisposable
{
public LofuBroken(ITestOutputHelper output) : base(output)
{
}
[Fact]
public async Task Test_1()
{
async Task test_method_1() {}
async Task test_method_2() {}
async Task test_method_3() {}
async Task test_method_4() {}
async Task test_method_5() {}
async Task test_method_6() {}
async Task test_method_7() { }
}
[Fact]
public async Task Test_2()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_3()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_4()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_5()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_6()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_7()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_8()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_9()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_10()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_11()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_12()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_13()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
[Fact]
public async Task Test_14()
{
async Task test_method_1() { }
async Task test_method_2() { }
async Task test_method_3() { }
async Task test_method_4() { }
async Task test_method_5() { }
async Task test_method_6() { }
async Task test_method_7() { }
}
public void Dispose()
{
}
}
Hi @Alastairm, thanks for reporting this issue.
I'll have to investigate this.
This is really strange.
I can confirm that this happened for me with LoFuUnit.Xunit and these versions:
<PackageReference Include="LoFuUnit.Xunit" Version="1.2.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">However, after I bump Microsoft.NET.Test.Sdk and xunit.runner.visualstudio this issue disappeared:
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">Now, even if I revert back to the previous versions, the execution is still in order.
Issue fixed, but I don't know how.
I could not reproduce this issue with NUnit or MSTest.
What versions do you use @Alastairm?
Can you try with the latest versions? Like here c34d67c
Thanks for the prompt reply @hlaueriksson, I've bumped Microsoft.NET.Test.Sdk to 16.10 (was 16.9.4), xunit.runner.visualstudio was already on 2.4.3 and the issue is still occurring.
I'm sorry to report that LoFuUnit cannot guarantee the order in which the test functions are executed.
The base class for tests invokes Type.GetMethods to find the local functions to use in a test method:
The docs from Microsoft states:
The GetMethods method does not return methods in a particular order, such as alphabetical or declaration order. Your code must not depend on the order in which methods are returned, because that order varies.
So, when the test functions happens to execute in the correct order you are just lucky.
I updated the docs:
The invocations will probably occur in the order that the test functions are declared, but it's not guaranteed. Do not write tests that depend on the order of test function execution.
@Alastairm thanks for the PR!
Version 1.3.0 of LoFuUnit is now released with your fix included 🥳