failed to wait for compliance caches

Question

failed to wait for compliance caches

yuriydzobak opened this issue 2 years ago · comments

Hi,
I don't have enabled the compliance in trivy operator

[INFO] ConfigAuditReports enabled
[INFO] VulnerabilityReports enabled
[INFO] RbacAssessmentReports enabled
[INFO] ExposedSecretReports enabled
[INFO] CISKubeBenchReports enabled
[INFO] InfraAssessmentReportClient enabled

but the application is crashed

Error: failed to wait for compliance caches to sync: timed out waiting for cache to be synced
Usage:
  trivy-operator-polr-adapter run [flags]

Flags:
  -c, --config string             target configuration file
      --enable-compliance         Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
      --enable-config-audit       Enable the transformation of ConfigAuditReports into PolicyReports
      --enable-exposed-secrets    Enable the transformation of ExposedSecretReports into PolicyReports
      --enable-infra-assessment   Enable the transformation of InfraAssessmentReports into PolicyReports
      --enable-kube-bench         Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
      --enable-rbac-assessment    Enable the transformation of RbacAssessmentReports into PolicyReports
      --enable-vulnerability      Enable the transformation of VulnerabilityReports into PolicyReports
  -h, --help                      help for run
  -k, --kubeconfig string         absolute path to the kubeconfig file

failed to wait for compliance caches to sync: timed out waiting for cache to be synced

Frank Jogeleit · Answer 1 · Wed Dec 28 2022 21:50:49 GMT+0800 (China Standard Time)

Hey, did you check if the clustercompliancereport CRD is installed in your cluster? Depending on your version of the trivy operator it is maybe not available.

Yurii Dzobak · Answer 2 · Wed Dec 28 2022 21:59:53 GMT+0800 (China Standard Time)

clustercompliancereport

it didn't install but i turned off in values.yaml but the app is still failing

    adapters:
      vulnerabilityReports:
        enabled: true
        # apply labels from the source report
        applyLabels: []
      configAuditReports:
        enabled: true
        applyLabels: []
      cisKubeBenchReports:
        enabled: true
        applyLabels: []
      complianceReports:
        enabled: false
        applyLabels: []
      rbacAssessmentReports:
        enabled: true
        applyLabels: []
      exposedSecretReports:
        enabled: true
        applyLabels: []
      infraAssessmentReports:
        enabled: true
        applyLabels: []

Frank Jogeleit · Answer 3 · Wed Dec 28 2022 22:02:54 GMT+0800 (China Standard Time)

Ah okay, then I misunderstood your question. I will have a look.

So it still tries to read compliancereports with complianceReports.enabled set to false?

Yurii Dzobak · Answer 4 · Wed Dec 28 2022 22:06:23 GMT+0800 (China Standard Time)

Ah okay, then I misunderstood your question. I will have a look.

You understood correct, I changed description, sorry

So it still tries to read compliancereports with complianceReports.enabled set to false?

yes, it's still

trivy-operator-adapter-5b849f8bc6-kml6s                           1/1     Running   3 (80s ago)     7m46s

[INFO] ConfigAuditReports enabled
[INFO] VulnerabilityReports enabled
[INFO] RbacAssessmentReports enabled
[INFO] ExposedSecretReports enabled
[INFO] CISKubeBenchReports enabled
[INFO] InfraAssessmentReportClient enabled
W1228 14:23:44.198888       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:44.198930       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope

W1228 14:23:45.660785       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:45.660814       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:23:46.585157       1 request.go:682] Waited for 1.001956396s due to client-side throttling, not priority and fairness, request: POST:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/namespaces/kubeshark/policyreports

W1228 14:23:48.513899       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:48.514308       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope



W1228 14:23:54.984785       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:23:54.984824       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:23:58.782402       1 request.go:682] Waited for 1.000297578s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/clusterpolicyreports/trivy-rbac-cpolr-clusterrole-679f75d6b5
W1228 14:24:06.809104       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:24:06.809133       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:24:14.785460       1 request.go:682] Waited for 1.003264357s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/wgpolicyk8s.io/v1alpha2/clusterpolicyreports/trivy-rbac-cpolr-clusterrole-strimzi-kafka-client
W1228 14:24:32.403984       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:24:32.404012       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:24:35.851365       1 request.go:682] Waited for 1.163116056s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
I1228 14:24:45.851449       1 request.go:682] Waited for 1.156883927s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/batch/v1beta1?timeout=32s
I1228 14:24:56.051268       1 request.go:682] Waited for 11.356236949s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/snapshot.storage.k8s.io/v1?timeout=32s
I1228 14:25:06.051717       1 request.go:682] Waited for 1.364532139s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/rbac.authorization.k8s.io/v1?timeout=32s
I1228 14:25:16.251319       1 request.go:682] Waited for 1.564216855s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/certificates.k8s.io/v1?timeout=32s
W1228 14:25:18.680478       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
E1228 14:25:18.680512       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1alpha1.InfraAssessmentReport: failed to list *v1alpha1.InfraAssessmentReport: infraassessmentreports.aquasecurity.github.io is forbidden: User "system:serviceaccount:monitoring:trivy-operator-adapter" cannot list resource "infraassessmentreports" in API group "aquasecurity.github.io" at the cluster scope
I1228 14:25:26.451744       1 request.go:682] Waited for 11.76386814s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/isindir.github.com/v1alpha2?timeout=32s
I1228 14:25:36.651437       1 request.go:682] Waited for 1.964975766s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/generators.external-secrets.io/v1alpha1?timeout=32s
Error: failed to wait for configaudit caches to sync: timed out waiting for cache to be synced
Usage:
  trivy-operator-polr-adapter run [flags]

Flags:
  -c, --config string             target configuration file
      --enable-compliance         Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
      --enable-config-audit       Enable the transformation of ConfigAuditReports into PolicyReports
      --enable-exposed-secrets    Enable the transformation of ExposedSecretReports into PolicyReports
      --enable-infra-assessment   Enable the transformation of InfraAssessmentReports into PolicyReports
      --enable-kube-bench         Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
      --enable-rbac-assessment    Enable the transformation of RbacAssessmentReports into PolicyReports
      --enable-vulnerability      Enable the transformation of VulnerabilityReports into PolicyReports
  -h, --help                      help for run
  -k, --kubeconfig string         absolute path to the kubeconfig file

failed to wait for configaudit caches to sync: timed out waiting for cache to be synced

Frank Jogeleit · Answer 5 · Thu Dec 29 2022 18:49:19 GMT+0800 (China Standard Time)

In this errors are the failed caches are configaudit, which are enabled. How large is your cluster? The only possible solution I could found for now would be to add a configuration for the timeout and increase it if needed. Could be possible that the default of 2 minutes are not enough for a higher amount of Trivy CRD Resources.

Yurii Dzobak · Answer 6 · Thu Dec 29 2022 18:59:56 GMT+0800 (China Standard Time)

I've disabled it too

helm get values trivy-operator-adapter                                                                                                                   dc06-ldap[12:57:39]
USER-SUPPLIED VALUES:
adapters:
  cisKubeBenchReports:
    enabled: true
  complianceReports:
    enabled: false
  configAuditReports:
    enabled: false
  exposedSecretReports:
    enabled: false
  infraAssessmentReports:
    enabled: false
  rbacAssessmentReports:
    enabled: false
  vulnerabilityReports:
    enabled: true
fullnameOverride: trivy-operator-adapter
nodeSelector:
  group-name: worker-group-infra
podAnnotations: {}
resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

Controlled By:  ReplicaSet/trivy-operator-adapter-5b849f8bc6
Containers:
  trivy-operator-polr-adapter:
    Container ID:   containerd://2fea4717def8ab8cae27f48f3acf1906a573b2de783c6f1461856ad4c70792fe
    Image:          ghcr.io/fjogeleit/trivy-operator-polr-adapter:0.3.1
    Image ID:       ghcr.io/fjogeleit/trivy-operator-polr-adapter@sha256:f270baae7515b261c82be5de39aac38c14d75c70bcf5cc1afb820b0ed91f265f
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 29 Dec 2022 12:54:28 +0200
      Finished:     Thu, 29 Dec 2022 12:56:29 +0200
    Ready:          False
    Restart Count:  16
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:        100m
      memory:     128Mi
    Readiness:    exec [/app/trivy-operator-polr-adapter version] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /app/config.yaml from config-file (ro,path="config.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s8gzt (ro)

trivy-operator-adapter-5b849f8bc6-5j75h                           1/1     Running   16 (6m51s ago)   91m

[INFO] VulnerabilityReports enabled
[INFO] CISKubeBenchReports enabled
I1229 10:48:17.221998       1 request.go:682] Waited for 1.100157662s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/kyverno.io/v1beta1?timeout=32s
I1229 10:48:27.421650       1 request.go:682] Waited for 1.300377222s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I1229 10:48:37.422086       1 request.go:682] Waited for 11.300079533s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/helm.toolkit.fluxcd.io/v2beta1?timeout=32s
I1229 10:48:47.622159       1 request.go:682] Waited for 1.501450994s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/events.k8s.io/v1?timeout=32s
I1229 10:48:57.622347       1 request.go:682] Waited for 1.501142727s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
I1229 10:49:07.823592       1 request.go:682] Waited for 11.702036764s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/core.strimzi.io/v1beta2?timeout=32s
I1229 10:49:18.022073       1 request.go:682] Waited for 1.901679701s due to client-side throttling, not priority and fairness, request: GET:https://10.11.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
Error: failed to wait for ciskubebench caches to sync: timed out waiting for cache to be synced
Usage:
  trivy-operator-polr-adapter run [flags]

Flags:
  -c, --config string             target configuration file
      --enable-compliance         Enable the transformation of ClusterComplianceDetailReports into ClusterPolicyReports
      --enable-config-audit       Enable the transformation of ConfigAuditReports into PolicyReports
      --enable-exposed-secrets    Enable the transformation of ExposedSecretReports into PolicyReports
      --enable-infra-assessment   Enable the transformation of InfraAssessmentReports into PolicyReports
      --enable-kube-bench         Enable the transformation of CISKubeBenchReports into ClusterPolicyReports
      --enable-rbac-assessment    Enable the transformation of RbacAssessmentReports into PolicyReports
      --enable-vulnerability      Enable the transformation of VulnerabilityReports into PolicyReports
  -h, --help                      help for run
  -k, --kubeconfig string         absolute path to the kubeconfig file

failed to wait for ciskubebench caches to sync: timed out waiting for cache to be synced

but the issue is still existing
The cluster is not so huge, just 140 PODs and 13 nodes

Frank Jogeleit · Answer 7 · Thu Dec 29 2022 19:35:34 GMT+0800 (China Standard Time)

Do you have the cis kube bench CRD installed? This CRD is not available in newer trivy operator versions.

Frank Jogeleit · Answer 8 · Thu Dec 29 2022 19:36:02 GMT+0800 (China Standard Time)

Does it also appear with only vulnerability reports enabled for example?

Yurii Dzobak · Answer 9 · Thu Dec 29 2022 21:18:13 GMT+0800 (China Standard Time)

I don't have cis kube
When i disabled it seems the application is working fine

trivy-operator-adapter-5b849f8bc6-vmzxz   1/1     Running   0          2m38s

$ klf trivy-operator-adapter-5b849f8bc6-vmzxz                                                                                                                                dc06-ldap[14:54:46]
[INFO] VulnerabilityReports enabled

Seems the issue was in this option
Thank you!

Yurii Dzobak · Answer 10 · Thu Dec 29 2022 21:20:38 GMT+0800 (China Standard Time)

I think, maybe disable the options in values.yaml and enable if it's necessary for user. That helps to reduce miss configuration
Also, I wish you Happy New Year!
Thanks

Frank Jogeleit · Answer 11 · Thu Dec 29 2022 21:32:43 GMT+0800 (China Standard Time)

Okay great. CIS Kube Bench is disabled by default and the readme has a deprecation info but I will improve it in the cli output as well.