failed to scan all layer contents: rhel: unable to create a mappingFile object
majewsky opened this issue · comments
Description of Problem / Feature Request
Since upgrading to Clair 4.6.0, we're sometimes seeing the indexing error failed to scan all layer contents: rhel: unable to create a mappingFile object pop up at random. This is not reproducible. Upon deleting the index report and indexing again, the error does not show up again.
One of the images in question that this happened on is index.docker.io/curlimages/curl@sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90 (the amd64 variant of the image index tagged as latest as of the time of this writing), so I recommend to use this image for testing. We only saw the unable to create a mappingFile object in one of our regional deployments (out of 15 regions), so that demonstrates the stochastic nature of the issue.
Expected Outcome
Indexing should not fail.
Actual Outcome
clair=# SELECT scan_result FROM indexreport WHERE manifest_id IN (SELECT id FROM manifest WHERE hash = 'sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90');
-[ RECORD 1 ]----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
scan_result | {"err": "failed to scan all layer contents: rhel: unable to create a mappingFile object", "state": "IndexError", "success": false, "packages": {}, "repository": {}, "environments": {}, "distributions": {}, "manifest_hash": "sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90"}
After deleting this index report and reindexing, we get the following index report:
{"err": "", "state": "IndexFinished", "success": true, "packages": {"1626": {"id": "1626", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libc-utils", "source": {"id": "1625", "cpe": "", "kind": "source", "name": "libc-dev", "version": "0.7.2-r3", "normalized_version": ""}, "version": "0.7.2-r3", "normalized_version": ""}, "207007": {"id": "207007", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-keys", "source": {"id": "207006", "cpe": "", "kind": "source", "name": "alpine-keys", "version": "2.4-r1", "normalized_version": ""}, "version": "2.4-r1", "normalized_version": ""}, "463765": {"id": "463765", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "apk-tools", "source": {"id": "463764", "cpe": "", "kind": "source", "name": "apk-tools", "version": "2.12.9-r3", "normalized_version": ""}, "version": "2.12.9-r3", "normalized_version": ""}, "463767": {"id": "463767", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "scanelf", "source": {"id": "463766", "cpe": "", "kind": "source", "name": "pax-utils", "version": "1.3.4-r0", "normalized_version": ""}, "version": "1.3.4-r0", "normalized_version": ""}, "529795": {"id": "529795", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "busybox", "source": {"id": "529794", "cpe": "", "kind": "source", "name": "busybox", "version": "1.35.0-r17", "normalized_version": ""}, "version": "1.35.0-r17", "normalized_version": ""}, "529801": {"id": "529801", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "ca-certificates-bundle", "source": {"id": "529800", "cpe": "", "kind": "source", "name": "ca-certificates", "version": "20220614-r0", "normalized_version": ""}, "version": "20220614-r0", "normalized_version": ""}, "529807": {"id": "529807", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "ssl_client", "source": {"id": "529794", "cpe": "", "kind": "source", "name": "busybox", "version": "1.35.0-r17", "normalized_version": ""}, "version": "1.35.0-r17", "normalized_version": ""}, "529809": {"id": "529809", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "zlib", "source": {"id": "529808", "cpe": "", "kind": "source", "name": "zlib", "version": "1.2.12-r3", "normalized_version": ""}, "version": "1.2.12-r3", "normalized_version": ""}, "571145": {"id": "571145", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-baselayout-data", "source": {"id": "571144", "cpe": "", "kind": "source", "name": "alpine-baselayout", "version": "3.2.0-r23", "normalized_version": ""}, "version": "3.2.0-r23", "normalized_version": ""}, "571147": {"id": "571147", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "alpine-baselayout", "source": {"id": "571144", "cpe": "", "kind": "source", "name": "alpine-baselayout", "version": "3.2.0-r23", "normalized_version": ""}, "version": "3.2.0-r23", "normalized_version": ""}, "629525": {"id": "629525", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "musl", "source": {"id": "629524", "cpe": "", "kind": "source", "name": "musl", "version": "1.2.3-r2", "normalized_version": ""}, "version": "1.2.3-r2", "normalized_version": ""}, "629527": {"id": "629527", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "musl-utils", "source": {"id": "629524", "cpe": "", "kind": "source", "name": "musl", "version": "1.2.3-r2", "normalized_version": ""}, "version": "1.2.3-r2", "normalized_version": ""}, "696381": {"id": "696381", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libcrypto1.1", "source": {"id": "696380", "cpe": "", "kind": "source", "name": "openssl", "version": "1.1.1t-r0", "normalized_version": ""}, "version": "1.1.1t-r0", "normalized_version": ""}, "696383": {"id": "696383", "cpe": "", "arch": "x86_64", "kind": "binary", "name": "libssl1.1", "source": {"id": "696380", "cpe": "", "kind": "source", "name": "openssl", "version": "1.1.1t-r0", "normalized_version": ""}, "version": "1.1.1t-r0", "normalized_version": ""}}, "repository": {}, "environments": {"1626": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "207007": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "463765": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "463767": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529795": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529801": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529807": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "529809": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49", "repository_ids": null, "distribution_id": "283"}], "571145": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "571147": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "629525": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "629527": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "696381": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}], "696383": [{"package_db": "lib/apk/db/installed", "introduced_in": "sha256:ea634e3b33ec1673331e65f2b19d3acef509af417267053d783e180c1c82af9f", "repository_ids": null, "distribution_id": "283"}]}, "distributions": {"283": {"id": "283", "cpe": "", "did": "alpine", "arch": "", "name": "Alpine Linux", "version": "3.16", "version_id": "", "pretty_name": "Alpine Linux v3.16", "version_code_name": ""}}, "manifest_hash": "sha256:17468885fb8a20cd6bc25316f8267492c4d758ba63a6838ce74b9a0ffe4d2e90"}
What is funny to me is that this is apparently an Alpine image, but the error indicates that it's related to rhel-specific code.
Environment
- Clair version/image: 4.6.0
- Clair client name/version: Keppel
- Host OS: RHEL 8.7 container on Flatcar 3374.2.4
- Kernel (e.g.
uname -a):Linux clair-indexer-64cd54fcb7-9dbzz 5.15.89-flatcar #1 SMP Wed Feb 15 18:00:42 -00 2023 x86_64 x86_64 x86_64 GNU/Linux - Kubernetes version (use
kubectl version): 1.25.6 - Network/Firewall setup: should not be relevant
Might be in a way link to the following commit :
commit e9f553e0dbe815203d012bcf3c23c4c2505d2cec
Author: crozzy <jXXXXX.cXXXXX@gmail.com>
Date: Thu Dec 15 15:04:58 2022 -0800
rhel: Check that after casting to mappingFile we have a usable mapper
Currently it is possible that if the repo2cpe_mapping_url or the
repo2cpe_mapping_file (or indeed if the endpoint is down) that we will
panic as the mappingFile will cast to a nil. This will check for a nil
mapper before it gets accessed and error out. This is also an issue
for name2repos_mapping_url and name2repos_mapping_file used by RHCC
scanner.
Signed-off-by: crozzy <jXXXXX.cXXXXXX@gmail.com>
What is funny to me is that this is apparently an Alpine image, but the error indicates that it's related to rhel-specific code.
Because we don't know anything about the image when we index it, all the (configured) scanners are run on every layer, hence why the rhel specific scanning is running.
The PR that holds the commit mentioned was to avert a panic in the above situation and instead surface. Since then we've changed the instantiation of a number of components and this should be non-issue going forward (quay/claircore#867) as the ingesting of the files should be a lot more infrequent.
This should be fixed in 4.7