Use names in UniqueIds of dynamic tests so that IDEs and humans can find dynamic tests easily

[k1w1m8]

Consider this trivial example dynamic test

package pkg;

import ...

public class JUnit5DynamicTest {

    @TestFactory
    Collection<DynamicNode> createDynamicTests() {

        List<DynamicNode> folder2 = new ArrayList<>();
        folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
        folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));

        List<DynamicNode> folder1 = new ArrayList<>();
        folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));

        List<DynamicNode> rootFolder = new ArrayList<>();
        rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
        return rootFolder;

    }

    private static class MyExecutable implements Executable {
        @Override
        public void execute() throws Throwable {
        }
    }
}

This creates a two level folder hierarchy with two tests in the child folder.

Once we run this test class in Intellij IDEA, we can then re-run an individual dynamic test by selecting it in the Run tab results treeview and clicking on 'Run'.

When we do this, IDEA creates a Run Configuration with a UniqueId field, which seems to contain the JUnit5 UniqueId class. It contains the following value (broken down one segment per line)

[engine:junit-jupiter]/
[class:pkg.JUnit5DynamicTest]/
[test-factory:createDynamicTests()]/
[dynamic-container:#1]/
[dynamic-container:#1]/
[dynamic-test:#1]

Problem is, the UniqueId value does not clearly identify the Dynamic Test instance that we are trying to run. This breaks the principle of least astonishment. The class and test factory/method are clearly identified by their names. However, the DynamicNodes seem to have a synthetic numeric identifier. What does this mean? And why do we not see the name we provided here?

Issues:

• The names of the dynamic nodes are not present, so we have no easy way to identify what dynamic test was run. The larger the number of tests in a container the worse the problem becomes.
• UniqueIds which contain synthetic numeric ids are presumably not stable over time, so if we insert or delete a dynamic test before this one, this Run configuration will select an incorrect dynamic test. This means the helpful Run Configuration feature of IDEA is crippled for Dynamic Tests

I think instead we should be using the supplied name here rather than synthetic ids. This makes it easy both for humans and IDEs to find individual dynamic tests easily.

This raises a question; what about names which are not unique? In general, I don't support the concept of duplicate names as I think it would make the reporting extremely confusing. I also doubt that many users would be creating duplicate names in their tests for this very reason. However, if it must be supported e.g. for backwards compatibility reasons, synthetic ids could be used only when required to enhance the names in case of name collisions (eg "name", "name" -> "name", "name#2").

Related

• #378

[k1w1m8]

Currently debugging into surefire-junit-platform for research into https://issues.apache.org/jira/browse/SUREFIRE-2147

Seems TestIdentifier for DynamicTests has a legacyReportingName of "createDynamicTests()[1][1][1]" which presumably should also be enhanced.

[Marc Philipp]

Problem is, the UniqueId value does not clearly identify the Dynamic Test instance that we are trying to run. This breaks the principle of least astonishment. The class and test factory/method are clearly identified by their names. However, the DynamicNodes seem to have a synthetic numeric identifier. What does this mean? And why do we not see the name we provided here?

The provided name is a display name and can contain any character so it would have to be sanitized in some way. Moreover, changing the display name should, ideally, not break re-running the existing IDE launch config. When renaming a test class/method, IDEs usually adapt existing launch configs but changing the display name is just typing in the editor. Therefore, I'm not convinced using display names in unique ID segments would be an improvement.

[k1w1m8]

Why would we need to sanitise the name?

I don't believe it is practical for an IDE to adapt a renamed launch config when JUnit has created the unique id using a synthetic id that only it is aware of. Certainly, IDEA does not do this.

Even if an IDE could manage to adapt renamed launch configs, it still leaves the problem that humans can't read the id and launch configs are broken in case of earlier additions or removals.

[M.P. Korstanje]

Problem is, the UniqueId value does not clearly identify the Dynamic Test instance that we are trying to run. This breaks the principle of least astonishment.

@k1w1m8

I feel I'm missing some context.

I get the impression, that you have a set of dynamic tests. Some are failing and you are trying to fix these one by one.

Rerunning one test can be done by selecting it from the UI, but when rerunning that test the UI loses the context with the other failing tests.

So to target the next failing test, all test must be ran again which is time consuming.The alternative is to edit the run configuration. And this provides only a UniqueId, unfit for human consumption.

Is that correct?

If so, I don't think the root of the problem lies with the UniqueId but rather with the IDE. Rerunning a failed test shouldn't lose cause a loss of context.

The JUnit Platform Launcher API provides a test plan prior to execution. There should be sufficient information there to allow an IDE to intelligently discard or retain the previous test run context (it could be displayed in a fainter color).

[Sam Brannen]

Unique IDs must by definition be "unique".

Including a user-defined display name (which has zero guarantee of being unique) could break the uniqueness, and we are not willing to do that.

Furthermore, unique IDs are based on technical aspects and are not meant to be human readable.

The class and test factory/method are clearly identified by their names.

That's correct, but they are the technical names. They are not the user-defined display names.

However, the DynamicNodes seem to have a synthetic numeric identifier. What does this mean?

Since there is no other way to identify dynamic tests by an implicit technical name, JUnit generates the IDs based on an incrementing counter which denotes the position within the stream.

In light of the above, I am closing this issue.

[k1w1m8]

Hmm, I fell into the usual trap of jumping ahead to a possible solution rather than focusing on outlining the problem clearly. Will I never learn?

This lead to people focusing on and rejecting the possible solution rather than focusing on the problem and how it might best be solved.

I'll raise a separate ticket outlining the problem in more detail and then hopefully we'll see if there is an acceptable solution.