validation summary

[Kevin Schneider]

We need a summary file that aggregates information about performed validations for consumption in downstream files.

Currently, this information was identified to be needed:

Field	Type	Optional	Description
HasCriticalFailures	boolean	NO	Indicates whether critical validation cases failed
CriticalTotal	integer	NO	total number of critical validation cases
CriticalPassed	integer	NO	number of critical validation cases passed
CriticalFailed	integer	NO	number of critical validation cases failed
HasNonCriticalFailures	boolean	NO	Indicates whether non-critical validation cases failed
NonCriticalTotal	integer	NO	total number of non-critical validation cases
NonCriticalPassed	integer	NO	number of non-critical validation cases passed
NonCriticalFailed	integer	NO	number of non-critical validation cases failed
ValidationPackageName	string	NO	The name of the validation package used for validation
ValidationPackageVersion	string	NO	The version of the validation package used for validation
ValidationPackageHookEndpoint	string	YES	An optional hook endpoint that a cqc backend can send events to

Upon initial discussions, this file should be of json format. We could additionally create a simple json schema for this.

Example

This example is the run summary of validation against the package test 2.0.1 with 5 critical and 5 non-critical validation cases, with 1 fail each:

{
    "HasCriticalFailures": true,
    "CriticalTotal": 5,
    "CriticalPassed": 4,
    "CriticalFailed": 1,
    "HasNonCriticalFailures": true,
    "NonCriticalTotal": 5,
    "NonCriticalPassed": 4,
    "NonCriticalFailed": 1,
    "ValidationPackageName": "test",
    "ValidationPackageVersion": "2.0.1",
    "ValidationPackageHookEndpoint": "https://my-arc-app.com/cqc-hook"
}

alternatively, we could nest 3 objects Critical, NonCritical, and ValidationPackage:

{
    "Critical": {
        "HasFailures": true,
        "Total": 5,
        "Passed": 4,
        "Failed": 1
    },
    "NonCritical": {
        "HasFailures": true,
        "Total": 5,
        "Passed": 4,
        "Failed": 1
    },
    "ValidationPackage": {
        "Name": "test",
        "Version": "2.0.1",
        "HookEndpoint": "https://my-arc-app.com/cqc-hook"
    }
}

[Kevin Schneider]

Additionally, we could also include the full package metadata in the ValidationPackage section/object

[j-bauer]

I prefer the nested structure personally. Doesn't really make the parsing any worse but it makes the data structure cleaner imo.

[Kevin Schneider]

Just a reference on what i settled on implementing in #99 in this regard:

{
    "Critical": {
        "HasFailures": false,
        "Total": 1,
        "Passed": 1,
        "Failed": 0,
        "Errored": 0
    },
    "NonCritical": {
        "HasFailures": false,
        "Total": 1,
        "Passed": 1,
        "Failed": 0,
        "Errored": 0
    },
    "ValidationPackage": {
        "Name": "test",
        "Version": "1.0.0",
        "Summary": "A package with CQC hook.",
        "Description": "A package with CQC hook. More text here.",
        "HookEndpoint": "http://test.com"
    }
}

With HookEndpoint bein optional, meaning key and value can be ommitted.

Also, we should specify this format in the arc specification.