Certstore system tests fail if certificates are not installed.
FAlbertDev opened this issue · comments
After building Botan on Windows, the Botan certstore system tests (src/tests/test_certstor_system.cpp
) failed on my system. The root certificate (D-TRUST Root Class 3 CA 2 EV 2009) was somehow not installed on my system, which caused this test to fail. The test implicitly asserts that some well-known CA certificates are installed. There are various options for how to handle this issue:
- We can ignore it and hope my system was the only one where the respective certificate was not installed. Maybe we could at least add a comment to the test that it fails if the certificate is not found on the system.
- We skip the test if the certificate is not found instead of failing it.
- Same as 2, but we add a flag to
botan-test
to avoid that we skip this test in CI.
In my opinion, it's not very urgent since no one has complained so far. It could be frustrating for a user when installing Botan, though.
The dependence on "certain" well-known root certificates in the test has always been somewhat an Achilles heel, unfortunately.
I'm in favor of (3), in a sense that we "somehow" flag the tests that assume the trust in certain root certs as "may-fail" in the test setup. AFAIK Botan's test system doesn't have this functionality, yet.
I'm envisioning something like:
auto test_that_assumes_existence_of_some_root() {
Test::Result result("Test OS certstor, MAY_FAIL);
/* ... test all the things ... */
return result
}
... if the test fails, it should print a warning by default but not set the return code of ./botan-test.exe
. Except the tests were invoked with ./botan-test.exe --strict
(or similar).