Resources limit and requests are added only to first container inside POD

[MichalGuzieniuk]

NRI adds limits and requests to resources section only to first container within POD. Second container resources are not affected.

Test case:

1. Run NRI with flag -honor-resources
2. Create foo and boo CRD networks

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: example.com/boo
  name: boo-network
  namespace: default
spec:
  config: '{"cniVersion": "0.3.0", "name": "boo-network", "type":"loopback"}'

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: example.com/foo
  name: foo-network
  namespace: default
spec:
  config: '{"cniVersion": "0.3.0", "name": "foo-network", "type":"loopback"}'

3. Create POD according to below specification

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: foo-network,boo-network
  name: nri-e2e-test
  namespace: default
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - sleep INF
    image: docker.io/alpine
    imagePullPolicy: Always
    name: test
  - command:
    - /bin/sh
    - -c
    - sleep INF
    image: docker.io/alpine
    imagePullPolicy: Always
    name: second-name
    resources:
      limits:
        example.com/foo: "9"
      requests:
        example.com/foo: "9"

4. Verify that resources in both containers are updated. Expected result
   First container

    resources:
      limits:
        example.com/boo: "1"
        example.com/foo: "1"
      requests:
        example.com/boo: "1"
        example.com/foo: "1"

Second container

    resources:
      limits:
        example.com/boo: "1"
        example.com/foo: "10"
      requests:
        example.com/boo: "1"
        example.com/foo: "10"

Current result:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: foo-network,boo-network
  name: nri-e2e-test
  namespace: default
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - sleep INF
    image: docker.io/alpine
    imagePullPolicy: Always
    name: test
    resources:
      limits:
        example.com/boo: "1"
        example.com/foo: "1"
      requests:
        example.com/boo: "1"
        example.com/foo: "1"
  - command:
    - /bin/sh
    - -c
    - sleep INF
    image: docker.io/alpine
    imagePullPolicy: Always
    name: second-name
    resources:
      limits:
        example.com/foo: "9"
      requests:
        example.com/foo: "9"

NRI logs

# kubectl logs -n kube-system network-resources-injector 
I0512 08:43:09.189322       1 main.go:69] starting mutating admission controller for network resources injection
I0512 08:43:09.190627       1 tlsutils.go:120] added client CA to cert pool from path '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
I0512 08:43:09.190651       1 tlsutils.go:122] added '1' client CA(s) to cert pool

[root@silpixa00397909 network-resources-injector]# kubectl apply -f pod.yaml 
pod/nri-e2e-test created

[root@silpixa00397909 network-resources-injector]# kubectl logs -n kube-system network-resources-injector 
I0512 09:02:57.435382       1 webhook.go:725] Received mutation request
I0512 09:02:57.441010       1 webhook.go:698] search v1.multus-cni.io/default-network in original pod annotations
I0512 09:02:57.441039       1 webhook.go:705] search v1.multus-cni.io/default-network in user-defined injections
I0512 09:02:57.441047       1 webhook.go:719] v1.multus-cni.io/default-network is not found in either pod annotations or user-defined injections
I0512 09:02:57.441059       1 webhook.go:698] search k8s.v1.cni.cncf.io/networks in original pod annotations
I0512 09:02:57.441069       1 webhook.go:701] k8s.v1.cni.cncf.io/networks is defined in original pod annotations
I0512 09:02:57.441096       1 webhook.go:257] 'foo-network,boo-network' is not in JSON format: invalid character 'o' in literal false (expecting 'a')... trying to parse as comma separated network selections list
I0512 09:02:57.473200       1 webhook.go:355] network attachment definition 'default/foo-network' found
I0512 09:02:57.473242       1 webhook.go:362] resource 'example.com/foo' needs to be requested for network 'default/foo-network'
I0512 09:02:57.481258       1 webhook.go:355] network attachment definition 'default/boo-network' found
I0512 09:02:57.481289       1 webhook.go:362] resource 'example.com/boo' needs to be requested for network 'default/boo-network'
I0512 09:02:57.481302       1 webhook.go:811] honor-resources=true
I0512 09:02:57.481345       1 webhook.go:821] injectHugepageDownApi=false
I0512 09:02:57.481377       1 webhook.go:879] patch after all mutations: [{add /spec/containers/0/resources/requests map[]} {add /spec/containers/0/resources/limits map[]} {add /spec/containers/0/resources/requests/example.com~1boo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/resources/limits/example.com~1boo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/resources/requests/example.com~1foo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/resources/limits/example.com~1foo {{1 0} {<nil>}  DecimalSI}} {add /spec/containers/0/volumeMounts/- {podnetinfo true /etc/podnetinfo  <nil> }} {add /spec/containers/1/volumeMounts/- {podnetinfo true /etc/podnetinfo  <nil> }} {add /spec/volumes/- {podnetinfo {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &DownwardAPIVolumeSource{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:annotations,FieldRef:&ObjectFieldSelector{APIVersion:,FieldPath:metadata.annotations,},ResourceFieldRef:nil,Mode:nil,},},DefaultMode:nil,} nil nil nil nil nil nil nil nil nil nil nil nil}}}]
I0512 09:02:57.481776       1 webhook.go:397] sending response to the Kubernetes API server

[Periyasamy Palanisamy]

@MichalGuzieniuk all containers in the pod share same network namespace, so I think adding device resource to first container itself would be sufficient.

[zshi]

@MichalGuzieniuk all containers in the pod share same network namespace, so I think adding device resource to first container itself would be sufficient.

Yes, this is true for kernel interface, every container inside Pod namespace would be able to see the attached sriov device, so it becomes less important on which container the resource is injected to.

@MichalGuzieniuk We do have a problem when the resource is userspace device (e.g. DPDK), in which case, the userspace device mounted in the first container is invisible to other containers (in the same pod). There is no solution to address this issue yet.

[Periyasamy Palanisamy]

@MichalGuzieniuk We do have a problem when the resource is userspace device (e.g. DPDK), in which case, the userspace device mounted in the first container is invisible to other containers (in the same pod). There is no solution to address this issue yet.

NRI mounts k8s.v1.cni.cncf.io/network-status annotation (which includes device-info) into all containers. wouldn't that suffice (example: to find out pci address of VF) for userspace device ?