Gloo + Istio integration 503
bewebi opened this issue · comments
Which tests failed?
Gloo + Istio integration tests Istio mTLS permissive peer auth when mtls is enabled for the upstream [It] should make a request with the expected cert header
Initial Investigation
Looks like pollution, with a previous test failing to clean up properly here
Deeper investigation needed
Additional Information
Logs here
Relevant section:
Fail handler msg Unexpected error:
<*errors.withStack | 0xc000b82df8>:
[kubectl delete -n istio-system peerauthentication test] failed: Error from server (NotFound): peerauthentications.security.istio.io "test" not found
: exit status 1
{
error: <*errors.withMessage | 0xc000c1b340>{
cause: <*exec.ExitError | 0xc000c1b320>{
ProcessState: {
pid: 52529,
status: 256,
rusage: {
Utime: {Sec: 0, Usec: 44791},
Stime: {Sec: 0, Usec: 20359},
Maxrss: 222676,
Ixrss: 0,
Idrss: 0,
Isrss: 0,
Minflt: 2092,
Majflt: 0,
Nswap: 0,
Inblock: 0,
Oublock: 0,
Msgsnd: 0,
Msgrcv: 0,
Nsignals: 0,
Nvcsw: 301,
Nivcsw: 29,
},
},
Stderr: nil,
},
msg: "[kubectl delete -n istio-system peerauthentication test] failed: Error from server (NotFound): peerauthentications.security.istio.io \"test\" not found\n",
},
stack: [0x62ed113, 0x62ec9cd, 0x62ec9a0, 0x690f9a5, 0x277b26f, 0x279a82c, 0x1fab561],
}
occurred
kube dump artifacts will be stored at: /home/runner/work/gloo/gloo/_output/kube2e-artifacts
kube dump artifacts will be stored at: /home/runner/work/gloo/gloo/_output/kube2e-artifacts/envoy-dump
Getting and storing envoy output for /config_dump path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:28:51 connecting to envoy failed with err Get "http://localhost:19000/config_dump": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /stats path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:28:51 connecting to envoy failed with err Get "http://localhost:19000/stats": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /clusters path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:28:51 connecting to envoy failed with err Get "http://localhost:19000/clusters": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /listeners path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:28:51 connecting to envoy failed with err Get "http://localhost:19000/listeners": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
S•Fail handler msg Timed out after 60.082s.
The function passed to Eventually failed at /home/runner/work/gloo/gloo/test/kube2e/helper/curl.go:124 with:
Expected
<*http.Response>: {
Status: <string>: ""
StatusCode: <int>: 503
Body: <string>: "no healthy upstream"
}
to have HTTP status
<int>: 200
expected: {
"StatusCode": 200,
"Body": {
"Substr": "\"X-Forwarded-Client-Cert\"",
"Args": null
},
"Headers": null,
"Custom": null
}
kube dump artifacts will be stored at: /home/runner/work/gloo/gloo/_output/kube2e-artifacts
kube dump artifacts will be stored at: /home/runner/work/gloo/gloo/_output/kube2e-artifacts/envoy-dump
Getting and storing envoy output for /config_dump path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:29:59 connecting to envoy failed with err Get "http://localhost:19000/config_dump": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /stats path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:30:00 connecting to envoy failed with err Get "http://localhost:19000/stats": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /clusters path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:30:00 connecting to envoy failed with err Get "http://localhost:19000/clusters": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
Getting and storing envoy output for /listeners path on gloo-proxy-http.gloo-system proxy
2024/06/12 16:30:00 connecting to envoy failed with err Get "http://localhost:19000/listeners": dial tcp [::1]:19000: connect: connection refused
Forwarding from 127.0.0.1:19000 -> 19000
Forwarding from [::1]:19000 -> 19000
Handling connection for 19000
------------------------------
• [FAILED] [62.419 seconds]
Gloo + Istio integration tests Istio mTLS permissive peer auth when mtls is enabled for the upstream [It] should make a request with the expected cert header
/home/runner/work/gloo/gloo/test/kube2e/istio/istio_integration_test.go:747
Timeline >>
No resources found in gloo-system namespace.
[FAILED] in [It] - /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.17.0/internal/node.go:472 @ 06/12/24 16:30:00.91
<< Timeline
[FAILED] Timed out after 60.082s.
The function passed to Eventually failed at /home/runner/work/gloo/gloo/test/kube2e/helper/curl.go:124 with:
Expected
<*http.Response>: {
Status: <string>: ""
StatusCode: <int>: 503
Body: <string>: "no healthy upstream"
}
to have HTTP status
<int>: 200
expected: {
"StatusCode": 200,
"Body": {
"Substr": "\"X-Forwarded-Client-Cert\"",
"Args": null
},
"Headers": null,
"Custom": null
}
In [It] at: /home/runner/go/pkg/mod/github.com/onsi/ginkgo/v2@v2.17.0/internal/node.go:472 @ 06/12/24 16:30:00.91
Occurred on #9606