Run tests of PHPUnit's own test suite in random order
staabm opened this issue · comments
I wonder whether the phpunits own test-suite should run in random order, to ease detection of inter-test dependencies or similar problems.
I tried it with
diff --git a/phpunit.xml b/phpunit.xml
index 05218b7c1..c65a4b53d 100644
--- a/phpunit.xml
+++ b/phpunit.xml
@@ -6,6 +6,7 @@
beStrictAboutOutputDuringTests="true"
failOnRisky="true"
failOnWarning="true"
+ executionOrder="default"
colors="true">
<testsuites>
<testsuite name="unit">and got some errors
➜ phpunit git:(infection) ✗ php ./phpunit
PHPUnit 11.1-g080cfb5ef by Sebastian Bergmann and contributors.
Runtime: PHP 8.2.17
Configuration: /Users/staabm/workspace/phpunit/phpunit.xml
Random Seed: 1711960388
...............................S............................. 61 / 3529 ( 1%)
............................................................. 122 / 3529 ( 3%)
.....................S....................................... 183 / 3529 ( 5%)
............................................................. 244 / 3529 ( 6%)
............................................................. 305 / 3529 ( 8%)
....F........................S............................... 366 / 3529 ( 10%)
....................................................F........ 427 / 3529 ( 12%)
................................................S............ 488 / 3529 ( 13%)
..............................................E.............. 549 / 3529 ( 15%)
............................................................. 610 / 3529 ( 17%)
............................................................. 671 / 3529 ( 19%)
............................................................. 732 / 3529 ( 20%)
............................................................. 793 / 3529 ( 22%)
............................................................. 854 / 3529 ( 24%)
............................................................. 915 / 3529 ( 25%)
............................................................. 976 / 3529 ( 27%)
............................................................. 1037 / 3529 ( 29%)
............................................................. 1098 / 3529 ( 31%)
............................................................. 1159 / 3529 ( 32%)
............................................................. 1220 / 3529 ( 34%)
............................................................. 1281 / 3529 ( 36%)
............................................................. 1342 / 3529 ( 38%)
............................................................. 1403 / 3529 ( 39%)
............................................................. 1464 / 3529 ( 41%)
............................................................. 1525 / 3529 ( 43%)
............................................................. 1586 / 3529 ( 44%)
............................................................. 1647 / 3529 ( 46%)
............................................................. 1708 / 3529 ( 48%)
............................................................. 1769 / 3529 ( 50%)
............................................................. 1830 / 3529 ( 51%)
............................................................. 1891 / 3529 ( 53%)
............................................................. 1952 / 3529 ( 55%)
............................................................. 2013 / 3529 ( 57%)
............................................................. 2074 / 3529 ( 58%)
............................................................. 2135 / 3529 ( 60%)
............................................................. 2196 / 3529 ( 62%)
............................................................. 2257 / 3529 ( 63%)
............................................................. 2318 / 3529 ( 65%)
............................................................. 2379 / 3529 ( 67%)
............................................................. 2440 / 3529 ( 69%)
............................................................. 2501 / 3529 ( 70%)
............................................................. 2562 / 3529 ( 72%)
............................................................. 2623 / 3529 ( 74%)
............................................................. 2684 / 3529 ( 76%)
............................................................. 2745 / 3529 ( 77%)
............................................................. 2806 / 3529 ( 79%)
............................................................. 2867 / 3529 ( 81%)
............................................................. 2928 / 3529 ( 82%)
............................................................. 2989 / 3529 ( 84%)
............................................................. 3050 / 3529 ( 86%)
............................................................. 3111 / 3529 ( 88%)
............................................................. 3172 / 3529 ( 89%)
............................................................. 3233 / 3529 ( 91%)
............................................................. 3294 / 3529 ( 93%)
............................................................. 3355 / 3529 ( 95%)
............................................................. 3416 / 3529 ( 96%)
............................................................. 3477 / 3529 ( 98%)
.................................................... 3529 / 3529 (100%)
Time: 00:46.600, Memory: 50.00 MB
There was 1 error:
1) PHPUnit\Framework\MockObject\ReturnValueGeneratorTest::test_Generates_test_stub_for_intersection_of_interfaces
PHPUnit\Framework\MockObject\RuntimeException: Return value for OriginalClassName::methodName() cannot be generated because the declared return type is an intersection, please configure a return value for this method
/Users/staabm/workspace/phpunit/src/Framework/MockObject/Runtime/ReturnValueGenerator.php:150
/Users/staabm/workspace/phpunit/tests/unit/Framework/MockObject/ReturnValueGeneratorTest.php:220
/Users/staabm/workspace/phpunit/tests/unit/Framework/MockObject/ReturnValueGeneratorTest.php:154
/Users/staabm/workspace/phpunit/src/Framework/TestCase.php:1173
/Users/staabm/workspace/phpunit/src/Framework/TestCase.php:482
/Users/staabm/workspace/phpunit/src/Framework/TestRunner.php:98
/Users/staabm/workspace/phpunit/src/Framework/TestCase.php:337
/Users/staabm/workspace/phpunit/src/Framework/TestSuite.php:361
/Users/staabm/workspace/phpunit/src/Framework/TestSuite.php:361
/Users/staabm/workspace/phpunit/src/Framework/TestSuite.php:361
/Users/staabm/workspace/phpunit/src/TextUI/TestRunner.php:62
/Users/staabm/workspace/phpunit/src/TextUI/Application.php:200
--
There were 2 failures:
1) /Users/staabm/workspace/phpunit/tests/end-to-end/cli/log-events-verbose-text.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
[00:00:00.045056083 / 00:00:00.000029208] [10485760 bytes] Test Preparation Started (PHPUnit\TestFixture\LogEventsText\Test::testExportObject)
[00:00:00.045097375 / 00:00:00.000041292] [10485760 bytes] Test Prepared (PHPUnit\TestFixture\LogEventsText\Test::testExportObject)
[00:00:00.047045375 / 00:00:00.001948000] [10485760 bytes] Test Failed (PHPUnit\TestFixture\LogEventsText\Test::testExportObject)
- Failed asserting that two variables reference the same object.
+ Failed asserting that two variables reference the same object.
[00:00:00.047126875 / 00:00:00.000081500] [10485760 bytes] Test Finished (PHPUnit\TestFixture\LogEventsText\Test::testExportObject)
[00:00:00.047213375 / 00:00:00.000086500] [10485760 bytes] Test Preparation Started (PHPUnit\TestFixture\LogEventsText\Test::testExportResource)
[00:00:00.047260500 / 00:00:00.000047125] [10485760 bytes] Test Prepared (PHPUnit\TestFixture\LogEventsText\Test::testExportResource)
[00:00:00.047412667 / 00:00:00.000152167] [10485760 bytes] Test Failed (PHPUnit\TestFixture\LogEventsText\Test::testExportResource)
- Failed asserting that two variables reference the same resource.
+ Failed asserting that two variables reference the same resource.
[00:00:00.047469417 / 00:00:00.000056750] [10485760 bytes] Test Finished (PHPUnit\TestFixture\LogEventsText\Test::testExportResource)
[00:00:00.047504583 / 00:00:00.000035166] [10485760 bytes] Test Suite Finished (PHPUnit\TestFixture\LogEventsText\Test, 7 tests)
[00:00:00.047533042 / 00:00:00.000028459] [10485760 bytes] Test Suite Finished (CLI Arguments, 7 tests)
/Users/staabm/workspace/phpunit/tests/end-to-end/cli/log-events-verbose-text.phpt:54
/Users/staabm/workspace/phpunit/src/Framework/TestSuite.php:361
/Users/staabm/workspace/phpunit/src/Framework/TestSuite.php:361
/Users/staabm/workspace/phpunit/src/TextUI/TestRunner.php:62
/Users/staabm/workspace/phpunit/src/TextUI/Application.php:200
is this something I should investigate?
is this something I should investigate?
If it tickles your fancy? Sure :)
Implemented in a86ed72.