Ignoring rule container-resources makes every rule exceptional
vajeen opened this issue · comments
Which version of kube-score are you using?
kube-score version: v1.16.1
What did you do?
Added
annotations:
kube-score/ignore: container-resources
And it made kube-score skip all the current issues
What did you expect to see?
See only issues related to container-resources to skip
What did you see instead?
kube-score passing all checks
Vajeen,
Just to confirm what you are reporting -- the ignore container-resources instruction should only disable the default container requests and limits resource requests, but instead, is disabling all container checks. Correct?
This can be confirmed by running the following --
$ cat score/testdata/pod-ephemeral-storage-annotation-ignore.yaml | ./kube-score score -vv -
v1/Pod pod-ephemeral-storage-annotation-ignore ✅
[SKIPPED] Stable version
· Skipped because stable-version is ignored
[SKIPPED] Label values
· Skipped because label-values is ignored
[SKIPPED] Container Image Tag
· Skipped because container-image-tag is ignored
[SKIPPED] Container Image Pull Policy
· Skipped because container-image-pull-policy is ignored
[SKIPPED] Container Ports Check
· Skipped because container-ports-check is ignored
[SKIPPED] Container Resource Requests Equal Limits
· Skipped because container-resource-requests-equal-limits is ignored
[SKIPPED] Container CPU Requests Equal Limits
· Skipped because container-cpu-requests-equal-limits is ignored
[SKIPPED] Container Ephemeral Storage Request Equals Limit
· Skipped because container-ephemeral-storage-request-equals-limit is ignored
[SKIPPED] Pod NetworkPolicy
· Skipped because pod-networkpolicy is ignored
[SKIPPED] Container Security Context User Group ID
· Skipped because container-security-context-user-group-id is ignored
[SKIPPED] Container Seccomp Profile
· Skipped because container-seccomp-profile is ignored
[SKIPPED] Pod Topology Spread Constraints
· Skipped because pod-topology-spread-constraints is ignored
[SKIPPED] Container Resources
· Skipped because container-resources is ignored
[SKIPPED] Environment Variable Key Duplication
· Skipped because environment-variable-key-duplication is ignored
[SKIPPED] Container Security Context ReadOnlyRootFilesystem
· Skipped because container-security-context-readonlyrootfilesystem is ignored
[SKIPPED] Container Memory Requests Equal Limits
· Skipped because container-memory-requests-equal-limits is ignored
[SKIPPED] Container Ephemeral Storage Request and Limit
· Skipped because container-ephemeral-storage-request-and-limit is ignored
[SKIPPED] Pod Probes
· Skipped because pod-probes is ignored
[SKIPPED] Container Security Context Privileged
· Skipped because container-security-context-privileged is ignored
Yes.
For example,
These are all the rules I want to skip
container-security-context-readonlyrootfilesystem,pod-networkpolicy,container-security-context-user-group-id,container-ephemeral-storage-request-and-limit,container-image-pull-policy,pod-probes,container-resources
If I take out container-resources skip rule, I get,
apps/v1/Deployment XXX in YYY 💥
[CRITICAL] Container Resources
· papi-server -> CPU limit is not set
Resource limits are recommended to avoid resource DDOS. Set
resources.limits.cpu
If I take out two two more (lets say the first two in the list -> container-security-context-readonlyrootfilesystem,pod-networkpolicy) I get this,
apps/v1/Deployment XXX in YYY 💥
[CRITICAL] Container Resources
· seal-trustweaver -> CPU limit is not set
Resource limits are recommended to avoid resource DDOS. Set
resources.limits.cpu
[CRITICAL] Container Security Context ReadOnlyRootFilesystem
· seal-trustweaver -> The pod has a container with a writable root filesystem
Set securityContext.readOnlyRootFilesystem to true
[CRITICAL] Pod NetworkPolicy
· The pod does not have a matching NetworkPolicy
Create a NetworkPolicy that targets this pod to control who/what
can communicate with this pod. Note, this feature needs to be
supported by the CNI implementation used in the Kubernetes cluster
to have an effect.
But if I add only container-resources skip rule, I get
apps/v1/Deployment XXX in YYY ✅
This is the output with -vv
apps/v1/Deployment XXX in YYY ✅
[SKIPPED] Stable version
· Skipped because stable-version is ignored
[SKIPPED] Label values
· Skipped because label-values is ignored
[SKIPPED] Container Ephemeral Storage Request Equals Limit
· Skipped because container-ephemeral-storage-request-equals-limit is ignored
[SKIPPED] Environment Variable Key Duplication
· Skipped because environment-variable-key-duplication is ignored
[SKIPPED] Container Security Context User Group ID
· Skipped because container-security-context-user-group-id is ignored
[SKIPPED] Container Security Context ReadOnlyRootFilesystem
· Skipped because container-security-context-readonlyrootfilesystem is ignored
[SKIPPED] Container Memory Requests Equal Limits
· Skipped because container-memory-requests-equal-limits is ignored
[SKIPPED] Container Seccomp Profile
· Skipped because container-seccomp-profile is ignored
[SKIPPED] Pod Probes
· Skipped because pod-probes is ignored
[SKIPPED] Container Image Tag
· Skipped because container-image-tag is ignored
[SKIPPED] Container Image Pull Policy
· Skipped because container-image-pull-policy is ignored
[SKIPPED] Container Ports Check
· Skipped because container-ports-check is ignored
[SKIPPED] Pod NetworkPolicy
· Skipped because pod-networkpolicy is ignored
[SKIPPED] Container Security Context Privileged
· Skipped because container-security-context-privileged is ignored
[SKIPPED] Container Resources
· Skipped because container-resources is ignored
[SKIPPED] Container Resource Requests Equal Limits
· Skipped because container-resource-requests-equal-limits is ignored
[SKIPPED] Container CPU Requests Equal Limits
· Skipped because container-cpu-requests-equal-limits is ignored
[SKIPPED] Container Ephemeral Storage Request and Limit
· Skipped because container-ephemeral-storage-request-and-limit is ignored
[SKIPPED] Deployment has host PodAntiAffinity
· Skipped because deployment-has-host-podantiaffinity is ignored
[SKIPPED] Deployment targeted by HPA does not have replicas configured
· Skipped because deployment-targeted-by-hpa-does-not-have-replicas-configured is ignored
[SKIPPED] Deployment Pod Selector labels match template metadata labels
· Skipped because deployment-pod-selector-labels-match-template-metadata-labels is ignored
[SKIPPED] Deployment has PodDisruptionBudget
· Skipped because deployment-has-poddisruptionbudget is ignored
I have some more testing to do, but believe I've got things working properly in my development environment.
The problem was related to a little snippet of code related to implied annotations related to container-resources. You'll note in https://github.com/zegl/kube-score/blob/master/README_CHECKS.md there are multiple checks related to container resources, but the container-resources check id refers only to the default cpu and memory checks. It does not, for example, include the default ephemeral storage checks. There was a previous request to include this check when the container-resources kube-score/ignore annotation was specified, thus eliminating the need to specify container-ephemeral-storage-request-and-limit as well.
Given the test file score/testdata/kube-score-ignore-annotations.yaml, we would expect all default container resource checks to be skipped, Let's test the assertion.
apiVersion: v1
kind: Pod
metadata:
name: pod-ephemeral-storage-annotation-ignore
annotations:
"kube-score/ignore": container-security-context-readonlyrootfilesystem,pod-networkpolicy,container-security-context-user-group-id,pod-probes,container-resources
spec:
containers:
- name: foobar
image: foo/bar:123
resources:
limits:
cpu: 200m
memory: 1Gi
ephemeral-storage: 2Gi
requests:
cpu: 200m
$ cat score/testdata/kube-score-ignore-annotations.yaml | ./kube-score score - -vv
v1/Pod pod-ephemeral-storage-annotation-ignore 💥
[OK] Stable version
[OK] Label values
[SKIPPED] Container Ephemeral Storage Request Equals Limit
· Skipped because container-ephemeral-storage-request-equals-limit is ignored
[SKIPPED] Container Security Context User Group ID
· Skipped because container-security-context-user-group-id is ignored
[SKIPPED] Container Seccomp Profile
· Skipped because container-seccomp-profile is ignored
[SKIPPED] Container CPU Requests Equal Limits
· Skipped because container-cpu-requests-equal-limits is ignored
[SKIPPED] Container Memory Requests Equal Limits
· Skipped because container-memory-requests-equal-limits is ignored
[CRITICAL] Container Image Pull Policy
· foobar -> ImagePullPolicy is not set to Always
It's recommended to always set the ImagePullPolicy to Always, to
make sure that the imagePullSecrets are always correct, and to
always get the image you want.
[SKIPPED] Pod Probes
· Skipped because pod-probes is ignored
[OK] Pod Topology Spread Constraints
· Pod Topology Spread Constraints
No Pod Topology Spread Constraints set, kube-scheduler defaults
assumed
[SKIPPED] Container Resource Requests Equal Limits
· Skipped because container-resource-requests-equal-limits is ignored
[OK] Environment Variable Key Duplication
[SKIPPED] Pod NetworkPolicy
· Skipped because pod-networkpolicy is ignored
[SKIPPED] Container Security Context ReadOnlyRootFilesystem
· Skipped because container-security-context-readonlyrootfilesystem is ignored
[SKIPPED] Container Resources
· Skipped because container-resources is ignored
[SKIPPED] Container Ports Check
· Skipped because container-ports-check is ignored
[OK] Container Security Context Privileged
[OK] Container Image Tag
[SKIPPED] Container Ephemeral Storage Request and Limit
· Skipped because container-ephemeral-storage-request-and-limit is ignoredWe now see the desired tests ignored and the rest run as desired.
I will run some additional tests before submitting the updates and issuing a PR for @zegl
Thanks @vajeen for a great bug report and @kmarteaux for debugging and fixing! 🌟