Question: Ctest vs gtest result is different
gjxo12 opened this issue · comments
Hello, I have one question like title.
It succeeds in Ctest, but fails in gtest.
this is private c++ code so I can only publish results....
the result is that Ctest is passed but htest is failed.. only one out of one hundred test case
Can you tell me more about the difference between ctest and gtest?
Below is part of Cmakelist.txt.
set(CMAKE_DEBUG_POSTFIX d)
if(GTEST)
include(FetchContent)
fetchcontent_declare(
googletest
URL https://github.com/google/googletest/archive/609281088cfefc76f9d0ce82e1ff6c30cc3591e5.zip
)
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
option(INSTALL_GTEST "Enable installation of googletest." OFF)
fetchcontent_makeavailable(googletest)
set_target_properties(
gtest gtest_main
PROPERTIES FOLDER "ExternalProjectTargets/googletest"
)
endif()
...
include(GoogleTest)
gtest_discover_tests(
${PROJECT_NAME}
EXTRA_ARGS --gtest_output=xml:file_name.xml
)
...
Thank you for the issue report. Could you provide a reproduction so we can investigate further?
Oh, I didn't know if this was an issue, so I asked a question.
As mentioned above, since the c++ project cannot be made public, it seems difficult to provide code that reproduces immediately.
So first I attach the results. It looks like the block below.
our c++ project jenkins result using Ctest and Gtest
Ctest run in cmkae build directory and Gtest run as binary
CTest
Start 185: TestCodeRefinery.ReplaceRegexAllOccurence
185/499 Test #185: TestCodeRefinery.ReplaceRegexAllOccurence ........................................... Passed 0.07 sec
Gtest
[ RUN ] TestCodeRefinery.ReplaceRegexAllOccurence
.../TestCodeRefinery.cpp:33: Failure
Expected equality of these values:
....
Which is: 2
0
....
1 error detected in the compilation of "/..../ReplaceRegexAllOccurence.cpp".
.....
[ FAILED ] TestCodeRefinery.ReplaceRegexAllOccurence (1 ms)
If this is an issue, I will make and provide a reproduction sample as soon as possible.
Finally, is there any difference between Ctest and Gtest?
GTest is a framework for writing unittests, and CTest is for running them, so they aren't really comparable. There are too many possible causes of the issue that you describe. We cannot look into any issues that we cannot reproduce. Please provide a reproduction.
A major difference is that ctest starts a new process for every test. If you have some uninitialized data somewhere this can make a difference.
@mjvankampen @derekmauro
Thank you for your kindness. I understand it to some extent.
maybe, I'll set a global locale variable or static member... and so on in the code with same test project
this will make difference.
lastly, Where can I refer to the things you mentioned..? can you tell me if possible..?