catkin test does not find unit test targets created by add_test since 0.8.0

[Shane Snover]

System Info

• Operating System: 5.4.0-147-generic #164-Ubuntu SMP Tue Mar 21 14:23:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: 3.8.10
• Version of catkin_tools: 0.8.0 (from pip)
• ROS Distro: noetic

Expected Behavior

We've been creating test targets with a combination of add_executable and add_test in our CMakeLists.txt files. We were using an older version of catkin_tools for a while and it successfully discovered the unit test binaries and executed them when invoking catkin test. It is expected that tests built this way are discovered and run.

Actual Behavior

After updating past catkin_tools 0.7, tests built this way are silently no longer discovered or run.

Steps to Reproduce the Issue

Create a ROS package with a unit test declared with add_executable and add_test. Using catkin_tools 0.7.2 or earlier, run catkin test and see tests are run (easier with a purposefully failing gtest). Update to catkin_tools 0.8.0 or later. Observe that tests are no longer run when catkin test is invoked.

Additional Information

I see that a native catkin test verb was introduced in v0.8.0 and it looks like previously the command was just a passthrough to another CLI. It looks like when this happened the behavior changed. We were able to get discovery to happen again using catkin_add_gtest. Incidentally, a member of my team was able to get the tests to run if invoked with catkin test --test-target test --continue-on-failure --verbose.

If it's just expected that ROS/catkin users are to use catkin_add_gtest, that's fair, but I couldn't find any documentation about the change and this issue will at least show up in searches. I suspect that larger ROS codebases could have a mix of the two approaches and it won't be as trivial to discover that certain unit tests are no longer running.