CTest returns non-zero when any tests fail, causing coverage to not finish
daviddoria opened this issue · comments
As stated in the script:
# NOTE! This should always have a ZERO as exit code
# otherwise the coverage generation will not complete.
However, the recommended way of running coverage analysis on all files is to make an add_custom_target that runs ctest. If any tests fail, ctest returns non-zero which causes the coverage analysis to abort. It seems reasonable to still run coverage even if there are failing tests - is there any way to get around this?
Yes this can be a problem. Unfortunately add_custom_target
always checks the exit code. Instead you could implement all COMMAND
-steps in a CMake-script file which is generated at configure time and run at build time (from the docs):
To run a full script, use the configure_file() command or the file(GENERATE) command to create it, and then specify a COMMAND to launch it
Sorry I decided not to implement this because it adds a lot of complexity (functionality split into several files, generated script files in build-directory) and we run coverage only when the tests passed. I leave this issue open in case somebody else comes across this.
I guess I don't understand which add_custom_target is doing any exit code checking? In your script you just run COMMAND ${Coverage_EXECUTABLE}
, so what difference does it make if that "fails" or succeeds? The files will still be there for the subsequent lcov calls to ingest, no?
In add_custom_target
you can run several COMMAND
s and if one of those return non-zero it simply stops execution of the remaining steps. There is no way around this. Another option would be to split into several add_custom_target
-calls.
I ended up with the following that seems to work. Thanks for your help!
add_custom_target(${Coverage_NAME}_cleanup
COMMAND ${LCOV_PATH} --directory . --zerocounters
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Old coverage cleanup..."
)
file(WRITE ${CMAKE_BINARY_DIR}/NoExitCodeTests.cmake "execute_process(COMMAND ctest)")
add_custom_target(${Coverage_NAME}_runtests
COMMAND ${CMAKE_COMMAND} -P NoExitCodeTests.cmake
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Running tests..."
VERBATIM
)
add_dependencies(${Coverage_NAME}_runtests ${Coverage_NAME}_cleanup)
add_custom_target(${Coverage_NAME}
# Capturing lcov counters and generating report
COMMAND ${LCOV_PATH} --directory . --capture --output-file ${Coverage_NAME}.info
COMMAND ${LCOV_PATH} --remove ${Coverage_NAME}.info ${COVERAGE_EXCLUDES} --output-file ${Coverage_NAME}.info.cleaned
COMMAND ${GENHTML_PATH} -o ${Coverage_NAME} ${Coverage_NAME}.info.cleaned
COMMAND ${CMAKE_COMMAND} -E remove ${Coverage_NAME}.info ${Coverage_NAME}.info.cleaned
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ${Coverage_DEPENDENCIES}
COMMENT "Processing code coverage counters and generating report."
)
add_dependencies(${Coverage_NAME} ${Coverage_NAME}_runtests)
The proposed solution looks quite complicated, so I have used a wrapper
setup_target_for_coverage(NAME coverage EXECUTABLE ${CMAKE_SOURCE_DIR}/ctestwrapper -j ${PROCESSOR_COUNT})
and the wrapper is just resetting the exit code:
#!/bin/sh ctest $@ exit 0
I faced the same issue and fixed it by changing
COMMAND ${Coverage_EXECUTABLE} ${Coverage_EXECUTABLE_ARGS}
to
COMMAND ${Coverage_EXECUTABLE} ${Coverage_EXECUTABLE_ARGS} || exit 0
so even if ctest or any other test driver return non zero, the above command will always return 0.
Tested with cmake v3.13.3