Are silent SKIPs useful or detrimental to the quality of testing?

[Jan Michalski]

QUESTION: Are silent SKIPs useful or detrimental to the quality of testing?

Facts

• Currently, we have several require_* calls that generate silent SKIPs
• Silent SKIPs were introduced to simplify eye-balling test logs. Since a smaller number of log lines are simpler to comprehend.
• Silent SKIPS are sometimes used to substitute for the lack of a mechanism allowing to ask for running only tests that fulfil a given conditions e.g. are meant to be run for a given CPU architecture.
• At the same time, silent SKIPs do not allow us to easily assess whether a given test was skipped rightfully or wrongly for the given configuration.

Discussion

In the case of a silent SKIP, the only observable fact is that the given record is missing. Note that you have to know what you are looking for to spot the given record is missing which imposes on the reader of the test logs an additional requirement of knowing what tests are meant to be run for the given configuration.

Possibly, this requirement is inevitable but till now it has not been accepted by our team.

Maybe silent SKIPs should be abandoned and instead, other mechanisms should be introduced to address the real issues silent SKIPs were used for.

Issues silent SKIPs address and alternative solutions for them

• selecting tests targeting given CPU architectures
  • #5854
• simplifying test logs
  • automatic test log processing instead of manual one?
• ...