[BUG] no-anti-affinity rules does not work as expected
day1118 opened this issue · comments
System info:
stackrox/kube-linter:0.2.6
Describe the bug
Even with anti-affinity rules apply to the pod, the no-anti-affinity rule fails
To Reproduce
# [Optional] Generate yaml manifest
helm repo add bitnami https://charts.bitnami.com/bitnami
helm template nginx bitnami/nginx --set replicaCount=2 --set containerSecurityContext.enabled=true --set containerSecurityContext.readOnlyRootFilesystem=true --set resources.requests.cpu=100m --set resources.limits.cpu=100m --set resources.requests.memory=100Mi --set resources.limits.memory=100Mi > temp/temp.yaml
# Run kube linter
docker run --rm -v ${PWD}/temp:/data stackrox/kube-linter:0.2.6 lint /data/temp.yamlSample YAML input
# Source: nginx/templates/deployment.yaml (stripped back)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
template:
metadata:
labels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
namespaces:
- "default"
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- name: nginx
image: docker.io/bitnami/nginx:1.21.6-debian-10-r0
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1001
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
Expected behavior
All rules should pass
Screenshots
KubeLinter 0.2.6
/data/temp.yaml: (object: <no namespace>/nginx apps/v1, Kind=Deployment) object has 2 replicas but does not specify inter pod anti-affinity (check: no-anti-affinity, remediation: Specify anti-affinity in your pod specification to ensure that the orchestrator attempts to schedule replicas on different nodes. Using podAntiAffinity, specify a labelSelector that matches pods for the deployment, and set the topologyKey to kubernetes.io/hostname. Refer to https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity for details.)Additional context
When replicas is set to 1, the rule correctly passes
The reason why kube-linter reports an error is because in podAffinityTerm you specified namespace and in pod spec not. If you add it to pod or remove from affinity then no error is reported.
# Source: nginx/templates/deployment.yaml (stripped back)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
template:
metadata:
+ namespace: default
labels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
namespaces:
- "default"
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- name: nginx
image: docker.io/bitnami/nginx:1.21.6-debian-10-r0
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1001
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi # Source: nginx/templates/deployment.yaml (stripped back)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
template:
metadata:
labels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
- namespaces:
- - "default"
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- name: nginx
image: docker.io/bitnami/nginx:1.21.6-debian-10-r0
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1001
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100MiI think the error message could be more specific and maybe include qualified name of affinity and pod so it will be easier to spot.
If we change func affinityTermMatchesLabelsAgainstNodes to return error instead of bool we can add more information to the user.
kube-linter/pkg/templates/antiaffinity/template.go
Lines 84 to 106 in e0525ba
For example with this patch we can have following errors:
+func affinityTermMatchesLabelsAgainstNodes(affinityTerm coreV1.PodAffinityTerm, podNamespace string, podLabels map[string]string, topologyKeyMatcher func(string) bool) error {
// If namespaces is not specified in the affinity term, that means the affinity term implicitly applies to the pod's namespace.
if len(affinityTerm.Namespaces) > 0 {
var matchingNSFound bool
@@ -92,15 +96,19 @@ func affinityTermMatchesLabelsAgainstNodes(affinityTerm coreV1.PodAffinityTerm,
}
}
if !matchingNSFound {
- return false
+ return fmt.Errorf("pod namespace %q not found in affinity namespaces [%s]", podNamespace, strings.Join(affinityTerm.Namespaces, ", "))
}
}
labelSelector, err := metaV1.LabelSelectorAsSelector(affinityTerm.LabelSelector)
if err != nil {
- return false
+ return err
}
- if topologyKeyMatcher(affinityTerm.TopologyKey) && labelSelector.Matches(labels.Set(podLabels)) {
- return true
+ if !topologyKeyMatcher(affinityTerm.TopologyKey) {
+ return fmt.Errorf("topology key matcher does not match %q", affinityTerm.TopologyKey)
}
- return false
+
+ if !labelSelector.Matches(labels.Set(podLabels)) {
+ return fmt.Errorf("pod labels %s does not match with lablel selector %s", labels.Set(podLabels).String(), labelSelector.String())
+ }
+ object has 2 replicas but does not specify inter pod anti-affinity: pod labels app.kubernetes.io/instance=nginx,app.kubernetes.io/name=nginx does not match with lablel selector app.kubernetes.io/instance=nginx1,app.kubernetes.io/name=nginx1"
object has 2 replicas but does not specify inter pod anti-affinity: pod namespace "" not found in affinity namespaces [default]
object has 2 replicas but does not specify inter pod anti-affinity: topology key matcher does not match "kubernetes.io/zone"
@day1118 Would you like to contribute a patch for this?
Thanks @janisz
I would have never worked this out.
To be clear, the patch you are proposing is to give a better error message, not to modify the logic right?
In this case, the template would have worked as expected - the namespace should be in the affinity rule (to prevent clashes with other namespaces) and the pod would have been deployed to the default namespace based on the kubectl/helm command used. The problem is just that kube-linter is not aware of this, correct?
Both correct. Indeed if you don't specify namespace in yaml then it could be specified in command but kube-linter has no knowledge how this command will be run so can't make any assumptions.
