Test scenarios fail for SCE-only rules if built without SCE
jan-cerny opened this issue · comments
Description of problem:
Automatus test scenarios for rule set_nftables_table fail when they are executed as a part of daily productization tests /per-rule/12/ansible and /per-rule/12/oscap. They give notchecked result instead of the expected results.
The reason is that the rule has only an SCE check and doesn't have an OVAL check. We don't build the content with SCE. SCE isn't included by default. Therefore, the notechecked is expected.
But the problem is how to handle this situation. We can waive it permanently, or we can introduce some logic for this situation, for example we can add a special test scenarios header.
SCAP Security Guide Version:
current upstream master as of 2024-05-27 as of HEAD 2f2a8c7
Operating System Version:
RHEL-9.4.0-20240526.24
Steps to Reproduce:
this is run inside contest
./automatus.py rule --libvirt qemu:///system contest --product rhel9 (--remediate-using ansible) ... set_nftables_table ...
Actual Results:
INFO - xccdf_org.ssgproject.content_rule_set_nftables_table
2024-05-28 09:47:41 test.py:126: running for rule: set_nftables_table
ERROR - Script nftables_incorrect_family.fail.sh using profile (all) found issue:
2024-05-28 09:47:47 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_incorrect_family.fail [/tmp/tmpgbnqb2tv/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_no_tables.fail.sh using profile (all) found issue:
2024-05-28 09:47:56 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_no_tables.fail [/tmp/tmpgb_3auju/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected fail during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
ERROR - Script nftables_table_present.pass.sh using profile (all) found issue:
2024-05-28 09:48:05 test.py:153: lib.results.report_plain:182: FAIL set_nftables_table/nftables_table_present.pass [/tmp/tmpiylbmqjp/out.txt]
ERROR - Rule evaluation resulted in notchecked, instead of expected pass during initial stage
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_set_nftables_table'.
Expected Results:
There shouldn't be any error. One of the tasks that we need to do is to determine what should be the expected results for this rule in this situation.
Additional Information/Debugging Steps:
The tests have been added by #11991.
adding to contest as a waiver: RHSecurityCompliance/contest#195