Robustness test flake

[Seth Heidkamp]

Which tests failed?

Robustness tests Updates Envoy endpoints, even if proxy is invalid works, even with deleted services [It] works, even with deleted services
  /home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:383

Initial Investigation

Different manifestation than #9251

Additional Information

https://github.com/solo-io/gloo/actions/runs/8477421605/job/23228320413

• [FAILED] [53.188 seconds]
Robustness tests Updates Envoy endpoints, even if proxy is invalid works, even with deleted services [It] works, even with deleted services
/home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:396

  Timeline >>
  STEP: assert we can route to svc1 @ 03/29/24 05:18:20.864
  STEP: assert we can not route to svc2 @ 03/29/24 05:18:28.553
  STEP: bounce gloo and envoy @ 03/29/24 05:18:33.785
  STEP: assert we can route to svc1 @ 03/29/24 05:18:38.871
  [FAILED] in [It] - /home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:383 @ 03/29/24 05:19:13.656
  << Timeline

  [FAILED] Timed out after 30.001s.
  The function passed to Eventually failed at /home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:367 with:
  Timed out after 30.001s.
  The function passed to Eventually returned the following error:
      <*errors.errorString | 0xc001fcf090>: 
      command terminated with exit code 7
       (exit status 7)
      {
          s: "command terminated with exit code 7\n (exit status 7)",
      }
  In [It] at: /home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:383 @ 03/29/24 05:19:13.656

  Full Stack Trace
    github.com/solo-io/gloo/test/kube2e/gateway_test.glob..func2.6.3.6()
    	/home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:383 +0x1ae
    github.com/solo-io/gloo/test/kube2e/gateway_test.glob..func2.6.3.8()
    	/home/runner/work/gloo/gloo/test/kube2e/gateway/robustness_test.go:410 +0x526
------------------------------
SSSSSSSSSSSSS

[Sam Heilbron]

I have moved this to blocked. I merged a PR that I believe should address this. If after 1 week, there are no more of these flakes, I will close it.