kube-apiserver log Err
lance5890 opened this issue · comments
What happened?
kube-apiserver log show many aggregated apiserver err as follows, and these log show kube-apiserver connect to openshift-apiserver has some problems
E0211 12:59:19.703718 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/templateinstances?allowWatchBookmarks=true&resourceVersion=52489&timeout=6m49s&timeoutSeconds=409&watch=true" audit-ID="b0a35b5f-17f7-44b4-bfa2-ff17525f9872"
E0211 12:59:19.722265 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/routes?allowWatchBookmarks=true&resourceVersion=52489&timeout=8m28s&timeoutSeconds=508&watch=true" audit-ID="ba21701a-5c36-4929-bd96-bdbadd309a87"
E0211 12:59:20.556152 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/build.openshift.io/v1/builds?allowWatchBookmarks=true&resourceVersion=58046&timeout=6m5s&timeoutSeconds=365&watch=true" audit-ID="19b6c3f9-38f3-4f9c-acd8-bf9413479ad3"
E0211 12:59:24.437018 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/image.openshift.io/v1/imagestreams?allowWatchBookmarks=true&resourceVersion=665503&timeout=6m2s&timeoutSeconds=362&watch=true" audit-ID="ab5cafa4-8934-4750-a166-67d88f7be662"
E0211 12:59:26.915270 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/apps.openshift.io/v1/deploymentconfigs?allowWatchBookmarks=true&resourceVersion=52506&timeout=8m24s&timeoutSeconds=504&watch=true" audit-ID="5d7fe004-24e4-4533-aec4-7f6ecbd0bbc4"
E0211 12:59:27.747743 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/brokertemplateinstances?allowWatchBookmarks=true&resourceVersion=52503&timeout=5m40s&timeoutSeconds=340&watch=true" audit-ID="bcea3626-0fc4-476b-a5a9-afbda0d4a546"
E0211 12:59:28.810936 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/build.openshift.io/v1/buildconfigs?allowWatchBookmarks=true&resourceVersion=52369&timeout=7m55s&timeoutSeconds=475&watch=true" audit-ID="fb96a4cb-9716-489e-9847-f2fa8121b081"
E0211 12:59:28.896780 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/oauth.openshift.io/v1/useroauthaccesstokens?allowWatchBookmarks=true&resourceVersion=52502&timeout=9m26s&timeoutSeconds=566&watch=true" audit-ID="bfb999c8-8bd8-4fca-a87a-c66a8fcbce1d"
E0211 12:59:28.991341 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/templates?allowWatchBookmarks=true&resourceVersion=52496&timeout=7m17s&timeoutSeconds=437&watch=true" audit-ID="906617cd-05be-4625-99f7-0825ae1cc4ea"
E0211 12:59:29.265106 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/security.openshift.io/v1/rangeallocations?allowWatchBookmarks=true&resourceVersion=52489&timeout=7m17s&timeoutSeconds=437&watch=true" audit-ID="3f058e9f-1a61-4aa7-9ee5-5ecd1eb6f1ac"
E0211 12:59:29.704176 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/template.openshift.io/v1/templateinstances?allowWatchBookmarks=true&resourceVersion=52489&timeout=9m0s&timeoutSeconds=540&watch=true" audit-ID="87173d21-a8b4-4408-bf46-5c4edf95509e"
E0211 12:59:29.722730 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/route.openshift.io/v1/routes?allowWatchBookmarks=true&resourceVersion=52489&timeout=5m45s&timeoutSeconds=345&watch=true" audit-ID="3b692bb1-13b4-48dc-9f8a-53280f66e371"
E0211 12:59:30.556852 19 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/build.openshift.io/v1/builds?allowWatchBookmarks=true&resourceVersion=58046&timeout=9m45s&timeoutSeconds=585&watch=true" audit-ID="3435b7d5-9249-48d3-8d64-d11976531462"
and the audit log will show as follows:
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"de6864a5-34e4-4bae-a087-16279d388d20","stage":"Panic","requestURI":"/apis/build.openshift.io/v1/builds?allowWatchBookmarks=true\u0026resourceVersion=735601\u0026timeout=8m20s\u0026timeoutSeconds=500\u0026watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.253.170.82"],"userAgent":"kube-controller-manager/v1.22.5 (linux/amd64) kubernetes/5236120/kube-controller-manager","objectRef":{"resource":"builds","apiGroup":"build.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"APIServer panic'd: net/http: abort Handler","reason":"InternalError","code":500},"requestReceivedTimestamp":"2023-02-13T07:19:38.273166Z","stageTimestamp":"2023-02-13T07:19:48.266617Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "system:kube-controller-manager" of ClusterRole "system:kube-controller-manager" to User "system:kube-controller-manager""}}
this is familiar with this https://bugzilla.redhat.com/show_bug.cgi?id=1922949
What did you expect to happen?
kube-apiserver do not show these errs
How can we reproduce it (as minimally and precisely as possible)?
without any operation, install ocp 4.9 , and install some service，the log will show like this
Anything else we need to know?
No response
Kubernetes version
$ kubectl version
# ocp 4.9 , k8s 1.22.5Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here