Fail to init - 500 server error when creating starboard-trivy-config ConfigMap

Question

Fail to init - 500 server error when creating starboard-trivy-config ConfigMap

KashifSaadat opened this issue 3 years ago · comments

What steps did you take and what happened:

Run the command: starboard init

Response: error: initializing Trivy plugin: an error on the server ("") has prevented the request from succeeding (post configmaps)

When running with increased log level verbosity, the following can be seen:

I1004 14:59:56.366142   17709 request.go:1181] Request Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"starboard","namespace":"starboard","creationTimestamp":null,"labels":{"app.kubernetes.io/managed-by":"starboard"}},"data":{"configAuditReports.scanner":"Polaris","kube-bench.imageRef":"docker.io/aquasec/kube-bench:0.6.3","kube-hunter.imageRef":"docker.io/aquasec/kube-hunter:0.6.1","kube-hunter.quick":"false","vulnerabilityReports.scanner":"Trivy"}}
I1004 14:59:56.366191   17709 round_trippers.go:435] curl -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: starboard/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://<eks-endpoint>/api/v1/namespaces/starboard/configmaps'
I1004 14:59:56.384582   17709 round_trippers.go:454] POST https://<eks-endpoint>/api/v1/namespaces/starboard/configmaps 201 Created in 18 milliseconds
I1004 14:59:56.384599   17709 round_trippers.go:460] Response Headers:
I1004 14:59:56.384604   17709 round_trippers.go:463]     Cache-Control: no-cache, private
I1004 14:59:56.384608   17709 round_trippers.go:463]     Content-Type: application/json
I1004 14:59:56.384611   17709 round_trippers.go:463]     Date: Mon, 04 Oct 2021 13:59:56 GMT
I1004 14:59:56.384614   17709 round_trippers.go:463]     Content-Length: 565
I1004 14:59:56.384617   17709 round_trippers.go:463]     Audit-Id: 5768f40b-e112-4996-a8ac-155770caf2d9
I1004 14:59:56.384635   17709 request.go:1181] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"starboard","namespace":"starboard","selfLink":"/api/v1/namespaces/starboard/configmaps/starboard","uid":"1ce2829d-a45d-46a3-9a3a-b5170055067c","resourceVersion":"154735579","creationTimestamp":"2021-10-04T13:59:56Z","labels":{"app.kubernetes.io/managed-by":"starboard"}},"data":{"configAuditReports.scanner":"Polaris","kube-bench.imageRef":"docker.io/aquasec/kube-bench:0.6.3","kube-hunter.imageRef":"docker.io/aquasec/kube-hunter:0.6.1","kube-hunter.quick":"false","vulnerabilityReports.scanner":"Trivy"}}
...
I1004 14:59:56.433642   17709 request.go:1179] Request Body:
00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 43 6f 6e 66  |k8s.....v1..Conf|
00000010  69 67 4d 61 70 12 f8 02  0a 5c 0a 16 73 74 61 72  |igMap....\..star|
00000020  62 6f 61 72 64 2d 74 72  69 76 79 2d 63 6f 6e 66  |board-trivy-conf|
00000030  69 67 12 00 1a 09 73 74  61 72 62 6f 61 72 64 22  |ig....starboard"|
00000040  00 2a 00 32 00 38 00 42  00 5a 29 0a 1c 61 70 70  |.*.2.8.B.Z)..app|
00000050  2e 6b 75 62 65 72 6e 65  74 65 73 2e 69 6f 2f 6d  |.kubernetes.io/m|
00000060  61 6e 61 67 65 64 2d 62  79 12 09 73 74 61 72 62  |anaged-by..starb|
00000070  6f 61 72 64 7a 00 12 30  0a 0e 74 72 69 76 79 2e  |oardz..0..trivy.|
00000080  69 6d 61 67 65 52 65 66  12 1e 64 6f 63 6b 65 72  |imageRef..docker|
00000090  2e 69 6f 2f 61 71 75 61  73 65 63 2f 74 72 69 76  |.io/aquasec/triv|
000000a0  79 3a 30 2e 31 39 2e 32  12 18 0a 0a 74 72 69 76  |y:0.19.2....triv|
000000b0  79 2e 6d 6f 64 65 12 0a  53 74 61 6e 64 61 6c 6f  |y.mode..Standalo|
000000c0  6e 65 12 22 0a 1a 74 72  69 76 79 2e 72 65 73 6f  |ne."..trivy.reso|
000000d0  75 72 63 65 73 2e 6c 69  6d 69 74 73 2e 63 70 75  |urces.limits.cpu|
000000e0  12 04 35 30 30 6d 12 25  0a 1d 74 72 69 76 79 2e  |..500m.%..trivy.|
000000f0  72 65 73 6f 75 72 63 65  73 2e 6c 69 6d 69 74 73  |resources.limits|
00000100  2e 6d 65 6d 6f 72 79 12  04 35 30 30 4d 12 24 0a  |.memory..500M.$.|
00000110  1c 74 72 69 76 79 2e 72  65 73 6f 75 72 63 65 73  |.trivy.resources|
00000120  2e 72 65 71 75 65 73 74  73 2e 63 70 75 12 04 31  |.requests.cpu..1|
00000130  30 30 6d 12 27 0a 1f 74  72 69 76 79 2e 72 65 73  |00m.'..trivy.res|
00000140  6f 75 72 63 65 73 2e 72  65 71 75 65 73 74 73 2e  |ources.requests.|
00000150  6d 65 6d 6f 72 79 12 04  31 30 30 4d 12 32 0a 0e  |memory..100M.2..|
00000160  74 72 69 76 79 2e 73 65  76 65 72 69 74 79 12 20  |trivy.severity. |
00000170  55 4e 4b 4e 4f 57 4e 2c  4c 4f 57 2c 4d 45 44 49  |UNKNOWN,LOW,MEDI|
00000180  55 4d 2c 48 49 47 48 2c  43 52 49 54 49 43 41 4c  |UM,HIGH,CRITICAL|
00000190  1a 00 22 00                                       |..".|
I1004 14:59:56.434377   17709 round_trippers.go:435] curl -v -XPOST  -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "Content-Type: application/vnd.kubernetes.protobuf" -H "User-Agent: starboard/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://<eks-endpoint>/api/v1/namespaces/starboard/configmaps'
I1004 14:59:56.481129   17709 round_trippers.go:454] POST https://<eks-endpoint>/api/v1/namespaces/starboard/configmaps 500 Internal Server Error in 46 milliseconds
I1004 14:59:56.481152   17709 round_trippers.go:460] Response Headers:
I1004 14:59:56.481157   17709 round_trippers.go:463]     Content-Length: 0
I1004 14:59:56.481161   17709 round_trippers.go:463]     Date: Mon, 04 Oct 2021 13:59:56 GMT
I1004 14:59:56.481180   17709 request.go:1181] Response Body:
error: initializing Trivy plugin: an error on the server ("") has prevented the request from succeeding (post configmaps)

Resources up until this point were created successfully, such as the api resources, namespace, starboard configmap and secret.

What did you expect to happen:

starboard init should complete successfully.

Environment:

Starboard versions tested: v0.11.0, v0.12.0
Kubernetes version: v1.17.17 (EKS)
OS: macOS 11.3

Daniel Pacak · Answer 1 · Tue Oct 05 2021 04:08:46 GMT+0800 (China Standard Time)

Internal Server Error indicates that there's some kind of a problem with K8s API server. I don't think it's caused by Starboard CLI in any way. Can you confirm that K8s API server pod is healthy and check its logs?

POST https://<eks-endpoint>/api/v1/namespaces/starboard/configmaps 500 Internal Server Error in 46 milliseconds

Kashif Saadat · Answer 2 · Tue Oct 05 2021 19:16:37 GMT+0800 (China Standard Time)

Hi @danielpacak. The Kube API Server is healthy and I can create resources without issue. Oddly there are no related error logs for the EKS apiserver, which I would have expected to see given it's a 500 Server Error response. You can see below that other resources were created successfully:

$ kubectl -n starboard get configmaps
NAME        DATA   AGE
starboard   5      21h

$ kubectl -n starboard get secrets
NAME                  TYPE                                  DATA   AGE
default-token-zndg7   kubernetes.io/service-account-token   3      21h
starboard             Opaque                                0      21h

$ kubectl api-resources --api-group aquasecurity.github.io
NAME                        SHORTNAMES           APIGROUP                 NAMESPACED   KIND
ciskubebenchreports         kubebench            aquasecurity.github.io   false        CISKubeBenchReport
clusterconfigauditreports   clusterconfigaudit   aquasecurity.github.io   false        ClusterConfigAuditReport
configauditreports          configaudit          aquasecurity.github.io   true         ConfigAuditReport
kubehunterreports           kubehunter           aquasecurity.github.io   false        KubeHunterReport
vulnerabilityreports        vuln,vulns           aquasecurity.github.io   true         VulnerabilityReport

The request body shown in the logs for the trivy configmap looks a little out of the ordinary, in comparison to all other requests that were made.

Daniel Pacak · Answer 3 · Fri Oct 08 2021 19:27:36 GMT+0800 (China Standard Time)

Thanks for the update @KashifSaadat I double checked that the error is not related to quite outdated version of K8s release 1.17.x. However it seems to be working fine with KIND cluster. I'll try to reproduce on EKS

$ kind-config kubectl get node -o wide
NAME                 STATUS   ROLES    AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
kind-control-plane   Ready    master   4m30s   v1.17.17   172.19.0.3    <none>        Ubuntu 21.04   5.10.47-linuxkit   containerd://1.5.2
kind-worker          Ready    <none>   3m56s   v1.17.17   172.19.0.2    <none>        Ubuntu 21.04   5.10.47-linuxkit   containerd://1.5.2

$ kubectl starboard version
Starboard Version: {Version:0.12.0 Commit:7b7db3acb673a7aaed50839aed168cba8163230a Date:2021-09-15T17:30:09Z}

$ kind-config kubectl starboard init -v 3
I1008 13:23:41.176111   31705 installer.go:377] Creating CRD "vulnerabilityreports.aquasecurity.github.io"
I1008 13:23:41.183962   31705 installer.go:377] Creating CRD "ciskubebenchreports.aquasecurity.github.io"
I1008 13:23:41.192490   31705 installer.go:377] Creating CRD "kubehunterreports.aquasecurity.github.io"
I1008 13:23:41.198269   31705 installer.go:377] Creating CRD "configauditreports.aquasecurity.github.io"
I1008 13:23:41.203089   31705 installer.go:377] Creating CRD "clusterconfigauditreports.aquasecurity.github.io"
I1008 13:23:41.208130   31705 installer.go:309] Creating Namespace "starboard"
I1008 13:23:41.226474   31705 installer.go:324] Creating ServiceAccount "starboard/starboard"
I1008 13:23:41.232070   31705 installer.go:341] Creating ClusterRole "starboard"
I1008 13:23:41.236432   31705 installer.go:359] Creating ClusterRoleBinding "starboard"

Daniel Pacak · Answer 4 · Mon Oct 11 2021 23:52:40 GMT+0800 (China Standard Time)

Just for the record, I cannot reproduce this error on my EKS cluster.

kubectl get node -o wide
NAME                                      STATUS   ROLES    AGE   VERSION               INTERNAL-IP   EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-10-0-6-80.us-west-2.compute.internal   Ready    <none>   16m   v1.17.11-eks-cfdc40   10.0.6.80     **.***.***.***   Amazon Linux 2   4.14.193-149.317.amzn2.x86_64   docker://19.3.6

$ kubectl starboard version
Starboard Version: {Version:0.12.0 Commit:7b7db3acb673a7aaed50839aed168cba8163230a Date:2021-09-15T17:30:09Z}

$ kubectl starboard init -v 3
I1011 17:46:56.995801   30789 installer.go:377] Creating CRD "vulnerabilityreports.aquasecurity.github.io"
I1011 17:46:57.388300   30789 installer.go:377] Creating CRD "ciskubebenchreports.aquasecurity.github.io"
I1011 17:46:58.152774   30789 installer.go:377] Creating CRD "kubehunterreports.aquasecurity.github.io"
I1011 17:46:58.535347   30789 installer.go:377] Creating CRD "configauditreports.aquasecurity.github.io"
I1011 17:46:58.950181   30789 installer.go:377] Creating CRD "clusterconfigauditreports.aquasecurity.github.io"
I1011 17:46:59.980854   30789 installer.go:309] Creating Namespace "starboard"
I1011 17:47:03.146610   30789 installer.go:324] Creating ServiceAccount "starboard/starboard"
I1011 17:47:04.849560   30789 installer.go:341] Creating ClusterRole "starboard"
I1011 17:47:05.560587   30789 installer.go:359] Creating ClusterRoleBinding "starboard"

$ kubectl describe cm -n starboard starboard-trivy-config
Name:         starboard-trivy-config
Namespace:    starboard
Labels:       app.kubernetes.io/managed-by=starboard
Annotations:  <none>

Data
====
trivy.severity:
----
UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
trivy.imageRef:
----
docker.io/aquasec/trivy:0.19.2
trivy.mode:
----
Standalone
trivy.resources.limits.cpu:
----
500m
trivy.resources.limits.memory:
----
500M
trivy.resources.requests.cpu:
----
100m
trivy.resources.requests.memory:
----
100M

BinaryData
====

Events:  <none>

$ kubectl starboard generate vulnerabilityreports deploy/nginx -v 3
I1011 17:47:38.514467   30825 scanner.go:68] Getting Pod template for workload: {Deployment nginx default}
I1011 17:47:39.389401   30825 scanner.go:79] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I1011 17:47:41.794572   30825 runner.go:79] Running task and waiting forever
I1011 17:47:41.795773   30825 runnable_job.go:74] Creating job "starboard/scan-vulnerabilityreport-74ddf5fb6"
I1011 17:47:42.470298   30825 reflector.go:219] Starting reflector *v1.Event (30m0s) from pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167
I1011 17:47:42.470305   30825 reflector.go:219] Starting reflector *v1.Job (30m0s) from pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167
I1011 17:47:42.470320   30825 reflector.go:255] Listing and watching *v1.Event from pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167
I1011 17:47:42.470324   30825 reflector.go:255] Listing and watching *v1.Job from pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167
I1011 17:47:42.664580   30825 runnable_job.go:130] Event: Created pod: scan-vulnerabilityreport-74ddf5fb6-frrb6 (SuccessfulCreate)
I1011 17:47:57.981347   30825 runnable_job.go:109] Stopping runnable job on task completion with status: Complete
I1011 17:47:57.981383   30825 runner.go:83] Stopping runner on task completion with error: <nil>
I1011 17:47:57.981393   30825 scanner.go:108] Scan job completed: starboard/scan-vulnerabilityreport-74ddf5fb6
I1011 17:47:57.981413   30825 scanner.go:199] Getting logs for nginx container in job: starboard/scan-vulnerabilityreport-74ddf5fb6
I1011 17:48:00.296373   30825 scanner.go:101] Deleting scan job: starboard/scan-vulnerabilityreport-74ddf5fb6

$ kubectl get vulnerabilityreports -o wide
NAME                     REPOSITORY      TAG    SCANNER   AGE   CRITICAL   HIGH   MEDIUM   LOW   UNKNOWN
deployment-nginx-nginx   library/nginx   1.16   Trivy     35s   25         85     83       15    0

Kashif Saadat · Answer 5 · Tue Oct 12 2021 20:30:37 GMT+0800 (China Standard Time)

Hi @danielpacak, thank you for the investigation. I looked into this further and found out it was due to an intermediary service attempting to evaluate and validate the request against a set of OPA policies, however it wasn't expecting a protobuf data format for the object. It failed to parse this (was expecting json, hence the other requests succeeding) and errored out.

Thanks for the help, happy for this issue to be closed as there's no requirement for any fixes or changes on starboard, given the protobuf data format is entirely valid.

Daniel Pacak · Answer 6 · Tue Oct 12 2021 20:44:48 GMT+0800 (China Standard Time)

Thank you for the update @KashifSaadat and I'm glad that you sorted it out 💪