'User defined injections' feature removes defined network annotations

[MichalGuzieniuk]

Problem:
NRI removes k8s.v1.cni.cncf.io/networks: foo-network label from annotations even if ConfigMap does not defines any network. In result, POD does not have additional network information.

Test steps:
As a precondition NRI is setup, and all networks are available.

1. Add network attachment definition
2. Add ConfigMap
3. Wait 1 minute - time needed to update POD with NRI
4. Create POD
   Expected:
   k8s.v1.cni.cncf.io/networks: foo-network annotation is available together with second entry in annotations list top-secret: password
   Current result:
   Network annotation is removed. Only new label from ConfigMap is visible.

Part of POD spec after modification

Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.244.1.2"
                    ],
                    "mac": "82:74:b7:10:75:94",
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.244.1.2"
                    ],
                    "mac": "82:74:b7:10:75:94",
                    "default": true,
                    "dns": {}
                }]
              top-secret: password

Expected POD spec

Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.244.2.3"
                    ],
                    "mac": "aa:9e:8a:a1:42:b6",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/foo-network",
                    "interface": "lo",
                    "ips": [
                        "127.0.0.1",
                        "::1"
                    ],
                    "mac": "00:00:00:00:00:00",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: foo-network
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.244.2.3"
                    ],
                    "mac": "aa:9e:8a:a1:42:b6",
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/foo-network",
                    "interface": "lo",
                    "ips": [
                        "127.0.0.1",
                        "::1"
                    ],
                    "mac": "00:00:00:00:00:00",
                    "dns": {}
                }]
              top-secret: password

POD, NAD, ConfigMap specs used in tests

apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-user-defined-injections
  namespace: kube-system
data:
  "customInjection": '{"op": "add", "path": "/metadata/annotations", "value": {"top-secret": "password"}}'

apiVersion: v1
kind: Pod
metadata:
  name: testpod
  labels:
    customInjection: "true"
  annotations:
    k8s.v1.cni.cncf.io/networks: foo-network
spec:
  containers:
  - name: app
    image: alpine
    command: [ "/bin/sh", "-c", "sleep INF" ]

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: foo-network
  annotations:
    k8s.v1.cni.cncf.io/resourceName: example.com/foo
spec:
  config: |
    {
      "cniVersion": "0.3.0",
      "name": "foo-network",
      "type": "loopback"
    }

[zshi]

@MichalGuzieniuk could you capture the network-resources-injector log when above issue happens?

[MichalGuzieniuk]

Logs after cluster initialization

I0409 12:10:02.188300       1 main.go:69] starting mutating admission controller for network resources injection
I0409 12:10:02.189260       1 tlsutils.go:120] added client CA to cert pool from path '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
I0409 12:10:02.189303       1 tlsutils.go:122] added '1' client CA(s) to cert pool

Additional log after ConfigMap consumption

I0409 12:12:32.252907       1 webhook.go:918] Initializing user-defined injections with key: customInjection, value: {"op": "add", "path": "/metadata/annotations", "value": {"top-secret": "password"}}

Logs after POD creation

I0409 12:14:04.363035       1 webhook.go:679] Received mutation request
I0409 12:14:04.367038       1 webhook.go:652] search v1.multus-cni.io/default-network in original pod annotations
I0409 12:14:04.367054       1 webhook.go:659] search v1.multus-cni.io/default-network in user-defined injections
I0409 12:14:04.367061       1 webhook.go:673] v1.multus-cni.io/default-network is not found in either pod annotations or user-defined injections
I0409 12:14:04.367072       1 webhook.go:652] search k8s.v1.cni.cncf.io/networks in original pod annotations
I0409 12:14:04.367079       1 webhook.go:655] k8s.v1.cni.cncf.io/networks is defined in original pod annotations
I0409 12:14:04.367098       1 webhook.go:256] 'foo-network' is not in JSON format: invalid character 'o' in literal false (expecting 'a')... trying to parse as comma separated network selections list
I0409 12:14:04.375919       1 webhook.go:354] network attachment definition 'default/foo-network' found
I0409 12:14:04.375948       1 webhook.go:361] resource 'example.com/foo' needs to be requested for network 'default/foo-network'
I0409 12:14:04.375962       1 webhook.go:765] honor-resources=false
I0409 12:14:04.375995       1 webhook.go:775] injectHugepageDownApi=true
I0409 12:14:04.376016       1 webhook.go:833] patch after all mutations: [{add /spec/containers/0/resources/requests map[]} {add /spec/containers/0/resources/limits map[]} {add /spec/containers/0/resources/requests/example.com~1foo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/resources/limits/example.com~1foo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/volumeMounts/- {podnetinfo false /etc/podnetinfo  <nil> }} {add /spec/volumes/- {podnetinfo {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &DownwardAPIVolumeSource{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:labels,FieldRef:&ObjectFieldSelector{APIVersion:,FieldPath:metadata.labels,},ResourceFieldRef:nil,Mode:nil,},DownwardAPIVolumeFile{Path:annotations,FieldRef:&ObjectFieldSelector{APIVersion:,FieldPath:metadata.annotations,},ResourceFieldRef:nil,Mode:nil,},},DefaultMode:nil,} nil nil nil nil nil nil nil nil nil nil nil nil}}} {add /metadata/annotations map[top-secret:password]}]
I0409 12:14:04.376500       1 webhook.go:396] sending response to the Kubernetes API server