[Flaking Test][sig-storage] should block a second pod from using an in-use ReadWriteOncePod volume on the same node
eddiezane opened this issue · comments
Which jobs are flaking?
- e2e-kops-aws-cni-cilium-eni
- ci-kubernetes-kind-e2e-parallel-1-29
- pull-kubernetes-e2e-kind
Which tests are flaking?
Kubernetes e2e suite: [It] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] read-write-once-pod [MinimumKubeletVersion:1.27] should block a second pod from using an in-use ReadWriteOncePod volume on the same node
Since when has it been flaking?
Unclear. It appears to be for a while but I haven't been able to find an issue already filed.
Testgrid link
https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-kind
Reason for failure (if possible)
[FAILED] failed to wait for FailedMount event for pod2: got error while getting events: client rate limiter Wait returned an error: context deadline exceeded
In [It] at: k8s.io/kubernetes/test/e2e/storage/testsuites/readwriteoncepod.go:233
[FAILED] failed to wait for FailedMount event for pod2: got error while getting events: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
Anything else we need to know?
Relevant SIG(s)
/sig storage
FYI: I used github action to do some tests on the job pull-kubernetes-e2e-kind, and it's always success if PARALLEL=false,the problem can only be reproduced when PARALLEL is true.
OutOfpods: Node didn't have enough resource: pods, requested: 1, used: 110, capacity: 110
STEP: Collecting events from namespace "read-write-once-pod-9757". @ 05/09/24 20:38:51.154
STEP: Found 10 events. @ 05/09/24 20:38:51.156
I0509 20:38:51.156989 65982 dump.go:53] At 2024-05-09 20:33:01 +0000 UTC - event for csi-hostpath5v7vm: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'csi-hostpath-read-write-once-pod-9757' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
I0509 20:38:51.157008 65982 dump.go:53] At 2024-05-09 20:33:20 +0000 UTC - event for csi-hostpath5v7vm: {csi-hostpath-read-write-once-pod-9757_csi-hostpathplugin-0_b225a3f7-b843-4de0-ac58-61bc09343ab8 } Provisioning: External provisioner is provisioning volume for claim "read-write-once-pod-9757/csi-hostpath5v7vm"
I0509 20:38:51.157018 65982 dump.go:53] At 2024-05-09 20:33:20 +0000 UTC - event for csi-hostpath5v7vm: {csi-hostpath-read-write-once-pod-9757_csi-hostpathplugin-0_b225a3f7-b843-4de0-ac58-61bc09343ab8 } ProvisioningSucceeded: Successfully provisioned volume pvc-aef51327-f502-4525-89bb-cb3f30838c95
I0509 20:38:51.157026 65982 dump.go:53] At 2024-05-09 20:33:21 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {default-scheduler } Scheduled: Successfully assigned read-write-once-pod-9757/pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff to kind-worker
I0509 20:38:51.157032 65982 dump.go:53] At 2024-05-09 20:33:22 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-aef51327-f502-4525-89bb-cb3f30838c95"
I0509 20:38:51.157039 65982 dump.go:53] At 2024-05-09 20:33:40 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {kubelet kind-worker} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.36.1-1" already present on machine
I0509 20:38:51.157046 65982 dump.go:53] At 2024-05-09 20:33:40 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {kubelet kind-worker} Created: Created container write-pod
I0509 20:38:51.157056 65982 dump.go:53] At 2024-05-09 20:33:42 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {kubelet kind-worker} Started: Started container write-pod
I0509 20:38:51.157063 65982 dump.go:53] At 2024-05-09 20:33:44 +0000 UTC - event for pod-da786ea3-645b-4ea1-9f62-51f97220b8be: {kubelet kind-worker} OutOfpods: Node didn't have enough resource: pods, requested: 1, used: 110, capacity: 110
I0509 20:38:51.157069 65982 dump.go:53] At 2024-05-09 20:38:44 +0000 UTC - event for pod-55dfb4dd-f05e-4776-a8e1-37283232c0ff: {kubelet kind-worker} Killing: Stopping container write-pod
/triage accepted