Nested tests fail in Java JUnit

Question

Nested tests fail in Java JUnit

cloudlena opened this issue 3 years ago · comments

Describe the bug
I'm using nested tests in Java using JUnit's Nested annotation: https://junit.org/junit5/docs/5.4.1/api/org/junit/jupiter/api/Nested.html. When running those tests with :Ultest or :UltestNearest, they fail. However, when I run them on the CLI with gradle test, they succeed. The same tests were working with ultest before I introduced the Nested pattern.

Logs

11:33:04 | INFO | MainThread | logging.py:create_logger:101 | Logger created
11:33:04 | DEBUG | MainThread | __init__.py:__init__:44 | Handler created
11:33:04 | INFO | MainThread | tracker.py:_init_test_file:97 | Initialising test file /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | __init__.py:_handle_coroutine:60 | Starting job with group update_positions
11:33:04 | INFO | MainThread | tracker.py:_async_update:57 | Updating positions in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(\\zs\\@Test\\s+\\ze)?%(\\zspublic\\s+\\ze)?void\\s+(\\w+)'], 'namespace': ['\\v^\\s*%(\\zspublic\\s+\\ze)?class\\s+(\\w+)']} to {'test': [re.compile('^\\s*(?:\\@Test\\s+)?(?:public\\s+)?void\\s+(\\w+)')], 'namespace': [re.compile('^\\s*(?:public\\s+)?class\\s+(\\w+)')]}
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test EntityTest-4281409786516624884 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test GetSpecificReservableBlock1255880266426898877 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test getsReservableBlock4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test worksWithSingleRange4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test worksWithNoSubnets4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test worksWithTwoSubnets4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test throwsOnNoMatchingRange4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresNotMatchingIpVersionRanges4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresInactiveRanges4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresNotMatchingAddressTypeRanges4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test throwsOnConflictingSubnet4178964679478858421 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test GetOptimalReservableBlock1255880266426898877 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test getsOptimalReservableBlock-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test worksWithSingleRange-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test worksWithIpVersion6-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresNotMatchingIpVersionRanges-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresInactiveRanges-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresNotMatchingAddressTypeRanges-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test ignoresRestrictedBlocks-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test throwsOnNoAvailability-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test with_managedByRange_case1-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_async_update:84 | New test with_managedByRange_case2-1318844551735082054 found in /home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java
11:33:04 | DEBUG | MainThread | tracker.py:_remove_old_positions:129 | No tests removed
11:33:04 | DEBUG | MainThread | __init__.py:_handle_coroutine:80 | Finished job with group update_positions
11:33:12 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java at line 26
11:33:12 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering getsReservableBlock4178964679478858421 as started
11:33:12 | DEBUG | MainThread | __init__.py:_handle_coroutine:60 | Starting job with group getsReservableBlock4178964679478858421
11:33:12 | DEBUG | MainThread | processes.py:run:50 | Starting test process getsReservableBlock4178964679478858421 with command ['gradle', 'test', '--tests', 'EntityTest.GetSpecificReservableBlock.getsReservableBlock'], cwd = /home/lena/customer/project, env = None
11:33:13 | DEBUG | MainThread | processes.py:run:76 | Process getsReservableBlock4178964679478858421 complete with exit code: 1
11:33:13 | DEBUG | MainThread | __init__.py:_register_result:299 | Registering getsReservableBlock4178964679478858421 as exited with result {"id": "getsReservableBlock4178964679478858421", "file": "/home/lena/customer/project/src/test/java/com/customer/prj/subprj/domain/entity/EntityTest.java", "code": 1, "output": "/tmp/ultestvzzm3fby/__home__lena__customer__project__src__test__java__com__customer__prj__subprj__domain__entity__EntityTest_java/getsReservableBlock4178964679478858421_out"}
11:33:13 | DEBUG | MainThread | __init__.py:_handle_coroutine:80 | Finished job with group getsReservableBlock4178964679478858421

To Reproduce
Create a Java test file and some tests with JUnit 5. The tests work correctly with ultest and succeed. Add a Nested class around the test cases, and they will fail.

Expected behavior
I expect the tests to succeed just like they do when running gradle test.

Additional context
Latest ultest plugin
Neovim 0.6.1
Alacritty 0.10.0
Arch Linux

Rónán Carrigan · Answer 1 · Sun Feb 06 2022 04:08:05 GMT+0800 (China Standard Time)

Can you tell me if they pass with vim-test's TestNearest? Does the output of the failed test tell you anything?

You can see the command being run is gradle test --tests EntityTest.GetSpecificReservableBlock.getsReservableBlock, does that look correct?

I have almost no familiarity with Java but from a very quick search I found this https://stackoverflow.com/questions/49180845/junit-test-in-nested-kotlin-class-not-found-when-running-gradle-test. Could it be that your test is not being found by gradle and when you run the CLI it's actually just not running the test at all and so passing?

Lena Fuhrimann · Answer 2 · Sun Feb 06 2022 19:07:01 GMT+0800 (China Standard Time)

Thanks, @rcarriga!

You're right. The vim-test command also fails:

 > Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> No tests found for given includes: [EntityTest.GetSpecificReservableBlock.getsReservableBlock](--tests filter)

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
5 actionable tasks: 2 executed, 3 up-to-date

[Process exited 1]

I'll open an issue there then 👍

Lena Fuhrimann · Answer 3 · Sun Feb 06 2022 19:22:22 GMT+0800 (China Standard Time)

I created vim-test/vim-test#627 respectively.