Coverage reports for asset tests are not generated in input packages
mrodm opened this issue · comments
Relates #1595
In #1595 , it is added support to create coverage reports for input packages. Even that, coverage report for asset tests it is not generated since it is not generated a list of assets for input packages.
This asset tests are based on kibana and elasticsearch assets. In case of input packages, there are no kibana assets, but there should be elasticsearch assets (e.g. index templates). Currently, the logic just takes into account generating that list based on data stream paths from the package. This is valid for integration packages but not for input packages.
elastic-package/internal/packages/assets.go
Lines 94 to 107 in b3f999e
@mrodm Should be a somewhat small fix, right? Tentatively putting this in the next sprint (or feel free to pull it ahead into this one if you'd like).
Double checking about this issue, AFAIK input packages should not have any kibana asset nor ingest pipelines asset. At least, according to the Package Spec for input packages it's not allowed (at least currently), any kibana nor elasticsearch/ingest_pipeline folders.
Should we try to check if there is any other specific asset in case of input packages ?
If there is no other asset to check, one option that it could be added here is to report this type of test (asset) as successful , if there is no assets loaded from the package directory expectedAssets (or there are no assets returned by Kibana installedPackage.Assets?).
diff --git internal/testrunner/runners/asset/runner.go internal/testrunner/runners/asset/runner.go
index d70ea19e..b0dc906b 100644
--- internal/testrunner/runners/asset/runner.go
+++ internal/testrunner/runners/asset/runner.go
@@ -118,6 +118,10 @@ func (r *runner) run() ([]testrunner.TestResult, error) {
return result.WithError(fmt.Errorf("could not load expected package assets: %w", err))
}
+ if len(expectedAssets) == 0 {
+ return result.WithSuccess()
+ }
+
results := make([]testrunner.TestResult, 0, len(expectedAssets))
for _, e := range expectedAssets {
rc := testrunner.NewResultComposer(testrunner.TestResult{In case of input packages, e.g. sql_input, there is no asset. And currently, it's reported as "No test results"
--- Test results for package: sql - START ---
No test results
--- Test results for package: sql - END ---
Done
And the coverage report has no line specified , no hits (so for SonarQube there is nothing):
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0" branch-rate="0" version="" timestamp="1707151537425395061" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0" complexity="0">
<sources></sources>
<packages>
<package name="sql" line-rate="0" branch-rate="0" complexity="0">
<classes></classes>
</package>
</packages>
</coverage>Returning that empty scenario as a success for test assets would give us:
- Test message:
--- Test results for package: sql - START ---
╭───────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ sql_input │ │ asset │ │ PASS │ 1.532703904s │
╰───────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: sql - END ---
- Coverage report with a positive hit for
assets:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0" branch-rate="0" version="" timestamp="1707150970289380816" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0" complexity="0">
<sources></sources>
<packages>
<package name="sql" line-rate="0" branch-rate="0" complexity="0">
<classes>
<class name="asset" filename="packages/sql/manifest.yml" line-rate="0" branch-rate="0" complexity="0">
<methods>
<method name="OK" signature="" line-rate="0" branch-rate="0" complexity="0">
<lines>
<line number="1" hits="1"></line>
</lines>
</method>
</methods>
<lines>
<line number="1" hits="1"></line>
</lines>
</class>
</classes>
</package>
</packages>
</coverage>Would that make sense for asset tests ? @jsoriano
Would it be better to just keep the same behaviour and create empty coverage reports for these packages?
Yes, I think we should generate empty reports for input packages. If they don't have assets there is nothing to report about them.
I've been trying to install some input packages from integrations repository to check about the assets that are installed (sql_input, jolokia_input and gcp_metrics).
In all of them, there are no assets in the corresponding tab:
Regarding Elasticsearch assets, if the input packages are installed using the API there is no asset for them:
However, if they are installed through the UI (adding them to an Agent policy), an index template is created:
Comparing with integration packages, when an integration package is installed using the API, an index template is created automatically (by Fleet).
No tested, but trying to install the input package and add that input package to an Agent policy in asset tests it would require at least:
- Create a new Agent policy
- Add that package to the Agent policy created.
- This would also require to add some configuration file as in system tests to fill the required fields if any (e.g.
gcp_metricshas a required field).
- This would also require to add some configuration file as in system tests to fill the required fields if any (e.g.
- Add a handler to remove the Agent policy created in (1).
Should this type of test check about Elasticsearsch assets too ? IIRC Other tests like pipeline or system tests would check that these assets already exist, is that right? And these resources/assets should be created by Fleet independently of the input package, is my understanding correct ?
Probably, we could keep just testing those ES assets that for integration packages as it is already in place, but not adding anything for input packages.
WDYT ? @jsoriano
If we agree on this, I will create a PR to remove the following comment, replacing it with an explanation:
elastic-package/internal/packages/assets.go
Line 156 in f025a06
Agree with not checking coverage of assets in input packages.
Example of (empty) reports created for input packages like sql_input:
-
no lines are reported in
manifest.yml, so no coverage at all. -
Cobertura:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE coverage SYSTEM "http://cobertura.sourceforge.net/xml/coverage-04.dtd">
<coverage line-rate="0" branch-rate="0" version="" timestamp="1707829422413503345" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0" complexity="0">
<sources></sources>
<packages>
<package name="sql" line-rate="0" branch-rate="0" complexity="0">
<classes></classes>
</package>
</packages>
</coverage>- Generic Test Data:
<?xml version="1.0" encoding="UTF-8"?>
<coverage version="1">
<!--Coverage for asset test--></coverage>