`[sig-cli] Kubectl client Simple pod should support inline execution and attach` flakes

[Sascha Grunert]

Which jobs are failing?

CRI-O e2e-conmonrs: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directory/pull-ci-cri-o-cri-o-main-ci-e2e-conmonrs

Which tests are failing?

[sig-cli] Kubectl client Simple pod should support inline execution and attach

Since when has it been failing?

Since ever, reproducible with latest conmon-rs on main.

Reason for failure (if possible)

The test flakes from time to time not containing the whole output in the logs:

  [FAILED] Expected
      <string>: read:stdin closed
      
  to contain substring
      <string>: read:value
  In [It] at: test/e2e/kubectl/kubectl.go:764 @ 01/04/23 15:09:48.618

https://github.com/kubernetes/kubernetes/blob/a66aad2d80dacc70025f95a8f97d2549ebd3208c/test/e2e/kubectl/kubectl.go#L757-L768

Anything else we need to know?

It looks like a race between the attach data and the processing in the stdout/err read loops. We added this code part to avoid such a race, but it will still happen after a couple of runs (even locally reproducible):

conmon-rs/conmon-rs/server/src/container_io.rs

Lines 337 to 343 in 29c5905

 _ = token.cancelled() => { 
     // Closing immediately may race with outstanding data on stdin for short lived 
     // containers. This means we try to read once again. 
     if let Ok(data) = attach.try_read() { 
         Self::handle_stdin_data(&data, &mut writer).await?; 
     } 
     return Ok(()); 

[Sascha Grunert]

I spent some days looking into this and run out of ideas how to solve it. Therefore I would appreciate another pair of eyes.

[Peter Hunt]

I am taking a look at this today