crash on startup after upgrading from 4.3.6 to 4.4.0
majewsky opened this issue · comments
Description of Problem / Feature Request
After upgrading from quay.io/projectquay/clair:4.3.6 to quay.io/projectquay/clair:4.4.0, Clair crashes on startup. The error message indicates a configuration error, but the configuration file is the same that used to work fine with 4.3.6. The configuration file is https://github.com/sapcc/helm-charts/blob/161e6637733f1d59a1e373b20fc1390d349e2a76/openstack/clair/templates/etc/_config.yaml.tpl. In that file, we interpolate secrets as described in the comment at the top, then we run clair -conf /tmp/clair-config.yaml -mode combo.
Expected Outcome
Clair should start up like it used to.
Actual Outcome
Logs from Clair:
{"level":"info","component":"main","version":"v4.4.0","time":"2022-03-18T10:39:12Z","message":"starting"}
{"level":"info","component":"main","lint":"automatically sizing number of concurrent requests (at $.indexer.index_report_request_concurrency)","time":"2022-03-18T10:39:12Z"}
{"level":"info","component":"main","lint":"this parameter will be ignored in a future release (at $.matcher.max_conn_pool)","time":"2022-03-18T10:39:12Z"}
{"level":"info","component":"main","lint":"no delivery mechanisms specified (at $.notifier)","time":"2022-03-18T10:39:12Z"}
{"level":"info","component":"main","time":"2022-03-18T10:39:12Z","message":"no CPU quota set, using default"}
{"level":"info","component":"main","cur":0,"prev":16,"time":"2022-03-18T10:39:12Z","message":"set GOMAXPROCS value"}
{"level":"info","component":"main","version":"v4.4.0","time":"2022-03-18T10:39:12Z","message":"ready"}
{"level":"info","component":"main","time":"2022-03-18T10:39:12Z","message":"launching introspection server"}
{"level":"info","component":"main","time":"2022-03-18T10:39:12Z","message":"launching http transport"}
{"level":"info","component":"main","time":"2022-03-18T10:39:12Z","message":"registered signal handler"}
{"level":"info","component":"initialize/Services","time":"2022-03-18T10:39:12Z","message":"begin service initialization"}
{"level":"warn","component":"introspection/New","time":"2022-03-18T10:39:12Z","message":"no health check configured; unconditionally reporting OK"}
{"level":"info","component":"introspection/Server.withPrometheus","endpoint":"/metrics","server":"0.0.0.0:8081","time":"2022-03-18T10:39:12Z","message":"configuring prometheus"}
{"level":"info","component":"introspection/New","time":"2022-03-18T10:39:12Z","message":"no distributed tracing enabled"}
{"level":"info","component":"libindex/New","time":"2022-03-18T10:39:12Z","message":"created database connection"}
{"level":"info","component":"libindex/New","time":"2022-03-18T10:39:12Z","message":"registered configured scanners"}
{"level":"info","component":"libvuln/New","count":32,"time":"2022-03-18T10:39:12Z","message":"initializing store"}
{"level":"info","component":"libvuln/New","time":"2022-03-18T10:39:12Z","message":"pool metrics already registered"}
{"level":"info","component":"libvuln/New","matchers":[{"name":"rhel","docs":"https://pkg.go.dev/github.com/quay/claircore/rhel"},{"name":"suse","docs":"https://pkg.go.dev/github.com/quay/claircore/suse"}],"time":"2022-03-18T10:39:12Z","message":"matchers created"}
{"level":"info","component":"rhel/Factory.Configure","time":"2022-03-18T10:39:12Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Factory.Configure","time":"2022-03-18T10:39:12Z","message":"configured HTTP client"}
{"level":"info","component":"libvuln/New","time":"2022-03-18T10:39:12Z","message":"libvuln initialized"}
{"level":"info","component":"libvuln/updates/Manager.Start","time":"2022-03-18T10:39:12Z","message":"starting initial updates"}
{"level":"info","component":"notifier/postgres/Init","time":"2022-03-18T10:39:12Z","message":"performing notifier migrations"}
{"level":"info","component":"notifier/service/New","interval":"1000h0m0s","time":"2022-03-18T10:39:12Z","message":"initializing poller"}
{"level":"info","component":"notifier/service/New","count":16,"time":"2022-03-18T10:39:12Z","message":"initializing processors"}
{"level":"info","component":"initialize/Services","reason":"no delivery mechanisms configured","time":"2022-03-18T10:39:12Z","message":"notifier disabled"}
{"level":"info","component":"initialize/Services","time":"2022-03-18T10:39:12Z","message":"end service initialization"}
{"level":"info","component":"httptransport/New","path":"/openapi/v1","time":"2022-03-18T10:39:12Z","message":"openapi discovery configured"}
{"level":"error","component":"libvuln/updates/Manager.Run","error":"Get \"https://access.redhat.com/security/data/oval/v2/PULP_MANIFEST\": context canceled","time":"2022-03-18T10:39:12Z","message":"failed constructing factory, excluding from run"}
{"level":"error","component":"main","error":"http transport configuration failed: could not configure notifier: NotifierMode requires a notifier service","time":"2022-03-18T10:39:12Z","message":"fatal error"}
Environment
- Clair version/image: 4.4.0
- Clair client name/version: N/A
- Host OS: Linux
- Kernel (e.g.
uname -a):Linux 5.10.102-flatcar #1 SMP Wed Mar 2 19:40:13 -00 2022 x86_64 GNU/Linux - Kubernetes version (use
kubectl version): 1.21 - Network/Firewall setup: irrelevant
Comparing with the startup log from 4.3.6, I can see this line being new:
{"level":"info","component":"main","lint":"no delivery mechanisms specified (at $.notifier)","time":"2022-03-18T10:39:12Z"}
I really hope that's not the fatal error here, because I deliberately don't have any notification mechanisms enabled. I don't use the notifier stuff at all.
Will look into this today. i think the changes with the config handling and the notifier didn't get matrix-tested with combo mode.
This should be fixed with #1531, and should be available to test in nightly-2022-03-22 tomorrow. I'll double-check and backport this week.
Thanks! I'll give that a try in our QA deployment tomorrow and let you know how it's looking.
nightly-2022-03-22 is looking good in our QA cluster. For reference, here's the startup part of the logs:
$ kubectl logs clair-6488cdbfdc-tq6th
{"level":"info","component":"main","version":"v4.4.0-9-g3273a969-dirty","time":"2022-03-22T09:43:17Z","message":"starting"}
{"level":"info","component":"main","lint":"automatically sizing number of concurrent requests (at $.indexer.index_report_request_concurrency)","time":"2022-03-22T09:43:17Z"}
{"level":"info","component":"main","lint":"this parameter will be ignored in a future release (at $.matcher.max_conn_pool)","time":"2022-03-22T09:43:17Z"}
{"level":"info","component":"main","lint":"no delivery mechanisms specified (at $.notifier)","time":"2022-03-22T09:43:17Z"}
{"level":"info","component":"main","time":"2022-03-22T09:43:17Z","message":"no CPU quota set, using default"}
{"level":"info","component":"main","cur":0,"prev":16,"time":"2022-03-22T09:43:17Z","message":"set GOMAXPROCS value"}
{"level":"info","component":"main","version":"v4.4.0-9-g3273a969-dirty","time":"2022-03-22T09:43:17Z","message":"ready"}
{"level":"info","component":"main","time":"2022-03-22T09:43:17Z","message":"registered signal handler"}
{"level":"info","component":"main","time":"2022-03-22T09:43:17Z","message":"launching introspection server"}
{"level":"info","component":"main","time":"2022-03-22T09:43:17Z","message":"launching http transport"}
{"level":"info","component":"initialize/Services","time":"2022-03-22T09:43:17Z","message":"begin service initialization"}
{"level":"warn","component":"introspection/New","time":"2022-03-22T09:43:17Z","message":"no health check configured; unconditionally reporting OK"}
{"level":"info","component":"introspection/Server.withPrometheus","endpoint":"/metrics","server":"0.0.0.0:8081","time":"2022-03-22T09:43:17Z","message":"configuring prometheus"}
{"level":"info","component":"introspection/New","time":"2022-03-22T09:43:17Z","message":"no distributed tracing enabled"}
{"level":"info","component":"libindex/New","time":"2022-03-22T09:43:17Z","message":"created database connection"}
{"level":"info","component":"libindex/New","time":"2022-03-22T09:43:17Z","message":"registered configured scanners"}
{"level":"info","component":"libvuln/New","count":32,"time":"2022-03-22T09:43:17Z","message":"initializing store"}
{"level":"info","component":"libvuln/New","time":"2022-03-22T09:43:17Z","message":"pool metrics already registered"}
{"level":"info","component":"libvuln/New","matchers":[{"name":"rhel","docs":"https://pkg.go.dev/github.com/quay/claircore/rhel"},{"name":"suse","docs":"https://pkg.go.dev/github.com/quay/claircore/suse"}],"time":"2022-03-22T09:43:17Z","message":"matchers created"}
{"level":"info","component":"rhel/Factory.Configure","time":"2022-03-22T09:43:17Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Factory.Configure","time":"2022-03-22T09:43:17Z","message":"configured HTTP client"}
{"level":"info","component":"libvuln/New","time":"2022-03-22T09:43:17Z","message":"libvuln initialized"}
{"level":"info","component":"libvuln/updates/Manager.Start","time":"2022-03-22T09:43:17Z","message":"starting initial updates"}
{"level":"info","component":"notifier/postgres/Init","time":"2022-03-22T09:43:17Z","message":"performing notifier migrations"}
{"level":"info","component":"notifier/service/New","interval":"1000h0m0s","time":"2022-03-22T09:43:17Z","message":"initializing poller"}
{"level":"info","component":"notifier/service/New","count":16,"time":"2022-03-22T09:43:17Z","message":"initializing processors"}
{"level":"info","component":"initialize/Services","reason":"no delivery mechanisms configured","time":"2022-03-22T09:43:17Z","message":"notifier disabled"}
{"level":"info","component":"initialize/Services","time":"2022-03-22T09:43:17Z","message":"end service initialization"}
{"level":"info","component":"httptransport/New","path":"/openapi/v1","time":"2022-03-22T09:43:17Z","message":"openapi discovery configured"}
{"level":"info","release":"disco","component":"ubuntu/Factory.UpdaterSet","status_code":404,"time":"2022-03-22T09:43:18Z","message":"ignoring release"}
{"level":"info","component":"ubuntu/Factory.UpdaterSet","release":"cosmic","status_code":404,"time":"2022-03-22T09:43:18Z","message":"ignoring release"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Updater.Configure","updater":"ubuntu-trusty-updater","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","updater":"ubuntu-xenial-updater","component":"ubuntu/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Updater.Configure","updater":"ubuntu-focal-updater","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Updater.Configure","updater":"ubuntu-eoan-updater","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"ubuntu/Updater.Configure","updater":"ubuntu-bionic-updater","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"debian/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"debian/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"debian/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"debian/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"debian/Updater.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
{"level":"info","component":"pkg/ovalutil/Fetcher.Configure","time":"2022-03-22T09:43:18Z","message":"configured HTTP client"}
...
I don't know if you usually close issues on fix-committed or on fix-released, so I'll leave that part to you.
Hi team, I am facing the same issue while deploying Clair in the distributed mode with Kubernetes (using minikube).
{"level":"error","component":"main","error":"http transport configuration failed: NotifierMode requires a notifier service","time":"2022-09-09T10:23:31Z","message":"fatal error"}
I used versions - “nightly, nightly-2022-08-27, 4.4.4, 4.4.0, 4.4.1”, and all showed the same error.
FWIW, I don't have this issue with 4.4.1 when running in combo mode. Is your crashing process in combo mode or notifier mode?
These are the logs from the notifier deployment in minikube.
Then it's definitely not this issue. This issue was about the combo-mode Clair validating its config as though it were in notifier mode. I suggest you open a separate issue for your problem.