`ClassOrderer.Random` and `MethodOrderer.Random` do not use same default seed
kurellajunior opened this issue · comments
The documentation is very misleading. It states, that the seed can be set individually for classes and methods, but neither configuration key is actually used:
ClassOrderer#RANDOM_SEED_PROPERTY_NAMEMethodOrderer#DEFAULT_ORDER_PROPERTY_NAME
instead for both random seeds is used MethodOrderer.Random#RANDOM_SEED_PROPERTY_NAME
This is also very inconsistent with the output of the tests which clearly suggests to set the respective seed to a specific value.
I do not know what is tha actual desired behaviour, so cannot provide a PR
The default order can be configured separately for classes and methods using the junit.jupiter.testclass.order.default and junit.jupiter.testmethod.order.default configuration parameters, respectively. Both ClassOrderer and MethodOrderer use the same junit.jupiter.execution.order.random.seed configuration parameter for customizing the seed used with their random orderers, though.
The documentation is very misleading. It states, that the seed can be set individually for classes and methods
Where does it state that?
neither configuration key is actually used
By "both" are you referring to the setting of the default class/method orderer or the custom seed?
This is also very inconsistent with the output of the tests which clearly suggests to set the respective seed to a specific value.
Which tests are you referring to?
The default order can be configured separately for classes and methods using the
junit.jupiter.testclass.order.defaultandjunit.jupiter.testmethod.order.defaultconfiguration parameters, respectively. BothClassOrdererandMethodOrdereruse the samejunit.jupiter.execution.order.random.seedconfiguration parameters for customizing the seed used with their random orderers, though.
that is understood
This is also very inconsistent with the output of the tests which clearly suggests setting the respective seed to a specific value.
Which tests are you referring to?
When I set the ClassOrderer$Random and MethodOrderer$Random as the test class and testmethod order default respectively, the tests log on the CONFIG level the (different) random seeds used. The assumption is they do so to replay a test run with the defined seeds. This is impossible, as both classes use the same configuration key as a custom seed
The documentation is very misleading. It states, that the seed can be set individually for classes and methods
Where does it state that?
Indeed now I see, it states here the opposite:
But actually uses two different seeds when running on fallback. This is then the inconsistency
Neither configuration key is actually used
By "both" are you referring to the setting of the default class/method orderer or the custom seed?
At least I could not find any documentation on how to use ClassOrderer#RANDOM_SEED_PROPERTY_NAME or MethodOrderer#DEFAULT_ORDER_PROPERTY_NAME. Luckily GitHub Copilot knew how to set those in the properties…
Thanks for the explanation, @kurellajunior. 👍
But actually uses two different seeds when running on fallback. This is then the inconsistency
That's an actual bug.
I've changed the title of this issue to reflect that, and I've assigned this to the 5.11 M2 milestone.