failing or missing response from external.metrics.k8s.io on GKE cluster

Question

failing or missing response from external.metrics.k8s.io on GKE cluster

TomoyukiSugiyama opened this issue a year ago · comments

Please suggestion how to use buildscaler on gke cluster.

I got a error with this command kubectl describe apiservices.apiregistration.k8s.io v1beta1.external.metrics.k8s.io

Status:
  Conditions:
    Last Transition Time:  2023-03-27T09:14:06Z
    Message:               failing or missing response from https://IP:6443/apis/external.metrics.k8s.io/v1beta1: Get "https://IP:6443/apis/external.metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available

When I checked behavior of buildscaler on local environment (docker desktop kubernetes version=v1.25.2), correctly worked.

Expected behavior

SUCCESS when type kubectl describe apiservices.apiregistration.k8s.io v1beta1.external.metrics.k8s.io

Actual behavior

FAILD when type kubectl describe apiservices.apiregistration.k8s.io v1beta1.external.metrics.k8s.io

Logs

% kubectl get apiservices.apiregistration.k8s.io                                                                               (git)-[master]
E0327 20:36:29.316717    2891 memcache.go:287] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0327 20:36:29.358895    2891 memcache.go:121] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0327 20:36:29.376263    2891 memcache.go:121] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0327 20:36:29.397670    2891 memcache.go:121] couldn't get resource list for external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAME                                      SERVICE                           AVAILABLE                  AGE
v1beta1.external.metrics.k8s.io           buildkite/buildscaler-apiserver   False (MissingEndpoints)   9s

Buildscaler version

v2.3.1

GKE version

v1.23.14-gke.1800

Platform

Google Cloud

Paweł Bojanowski · Answer 1 · Mon Mar 27 2023 20:51:00 GMT+0800 (China Standard Time)

Hi @TomoyukiSugiyama , thanks for posting detailed description of your problem with versions, it was helpful in re-producing the very same environment.

I tried this on GKE v1.23.14 and Buildscaler 2.3.1 and I don't see the same issue as you.
Could you verify if:

buildscaler-apiserver deployment is up and running

(you can check it using kubectl get pods -n <namespace-where-you-deployed-buildscaler> -l app=buildscaler-apiserver. If it is running, can you post here the output of kubectl logs -n <namespace-where-you-deployed-buildscaler> deployment/buildscaler-apiserver ?

From the logs you posted it seems that k8s service buildscaler-apiserver has no endpoints, which may indicate that buildscaler-apiserver deployment is not running.

thanks

Tomoyuki Sugiyama · Answer 2 · Mon Mar 27 2023 22:36:37 GMT+0800 (China Standard Time)

Hi @hidalgopl
Thanks for your reply.

I got a output of kubectl logs -n buildkite deployment/buildscaler-apiserver
and I got following error.
E0327 14:16:04.445773 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)

logs

I0327 14:16:03.768093       1 main.go:95] using buildkite scraper & metrics provider
I0327 14:16:03.768298       1 buildkite.go:198] Collecting agent metrics for all queues
I0327 14:16:04.216841       1 serving.go:341] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0327 14:16:04.217330       1 dynamic_serving_content.go:110] "Loaded a new cert/key pair" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0327 14:16:04.432696       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 12 milliseconds
I0327 14:16:04.433300       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I0327 14:16:04.434067       1 config.go:679] Not requested to run hook priority-and-fairness-config-consumer
I0327 14:16:04.435293       1 healthz.go:174] Installing health checkers for (/healthz): "ping","log","poststarthook/max-in-flight-filter"
I0327 14:16:04.435465       1 healthz.go:174] Installing health checkers for (/livez): "ping","log","poststarthook/max-in-flight-filter"
I0327 14:16:04.435635       1 healthz.go:174] Installing health checkers for (/readyz): "ping","log","poststarthook/max-in-flight-filter","shutdown"
I0327 14:16:04.435835       1 genericapiserver.go:419] [graceful-termination] ShutdownTimeout=1m0s
I0327 14:16:04.440404       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 4 milliseconds
I0327 14:16:04.440682       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0327 14:16:04.440741       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0327 14:16:04.440758       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0327 14:16:04.440763       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0327 14:16:04.440786       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0327 14:16:04.440812       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0327 14:16:04.440864       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.440992       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.441045       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0327 14:16:04.441027       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" certDetail="\"localhost@1679926564\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1679926564\" (2023-03-27 13:16:03 +0000 UTC to 2024-03-26 13:16:03 +0000 UTC (now=2023-03-27 14:16:04.440999379 +0000 UTC))"
I0327 14:16:04.441326       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1679926564\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1679926564\" (2023-03-27 13:16:04 +0000 UTC to 2024-03-26 13:16:04 +0000 UTC (now=2023-03-27 14:16:04.441303109 +0000 UTC))"
I0327 14:16:04.440962       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.441482       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.441576       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0327 14:16:04.440865       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.441641       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:04.441427       1 secure_serving.go:266] Serving securely on [::]:6443
I0327 14:16:04.441742       1 genericapiserver.go:378] [graceful-termination] waiting for shutdown to be initiated
I0327 14:16:04.443073       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0327 14:16:04.443238       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0327 14:16:04.444180       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 2 milliseconds
I0327 14:16:04.445435       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787&timeout=9m42s&timeoutSeconds=582&watch=true 403 Forbidden in 1 milliseconds
I0327 14:16:04.445707       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787&timeout=8m19s&timeoutSeconds=499&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:04.445773       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
E0327 14:16:04.445883       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:04.445921       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787&timeout=8m1s&timeoutSeconds=481&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:04.445957       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:04.475369       1 buildkite.go:155] Found organization "MYORG"
I0327 14:16:04.475737       1 buildkite.go:358] Buildkite > Org=MYORG > TotalAgentCount=31
I0327 14:16:04.475755       1 buildkite.go:358] Buildkite > Org=MYORG > BusyAgentPercentage=9
I0327 14:16:04.475772       1 buildkite.go:358] Buildkite > Org=MYORG > ScheduledJobsCount=2
I0327 14:16:04.475777       1 buildkite.go:358] Buildkite > Org=MYORG > RunningJobsCount=3
I0327 14:16:04.475781       1 buildkite.go:358] Buildkite > Org=MYORG > UnfinishedJobsCount=15
I0327 14:16:04.475785       1 buildkite.go:358] Buildkite > Org=MYORG > WaitingJobsCount=10
I0327 14:16:04.475789       1 buildkite.go:358] Buildkite > Org=MYORG > IdleAgentCount=28
I0327 14:16:04.475793       1 buildkite.go:358] Buildkite > Org=MYORG > BusyAgentCount=3
I0327 14:16:04.475801       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue1 > IdleAgentCount=1
I0327 14:16:04.475806       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue1 > BusyAgentCount=0
I0327 14:16:04.475810       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue1 > TotalAgentCount=1
I0327 14:16:04.475814       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue1 > BusyAgentPercentage=0
I0327 14:16:04.475818       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue2 > BusyAgentCount=0
I0327 14:16:04.475826       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue2 > TotalAgentCount=5
I0327 14:16:04.475830       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue2 > BusyAgentPercentage=0
I0327 14:16:04.475834       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue2 > IdleAgentCount=5
I0327 14:16:04.475838       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue3 > IdleAgentCount=3
I0327 14:16:04.475842       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue3 > BusyAgentCount=0
I0327 14:16:04.475850       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue3 > TotalAgentCount=3
I0327 14:16:04.475854       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue3 > BusyAgentPercentage=0
I0327 14:16:04.475859       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue4 > IdleAgentCount=3
I0327 14:16:04.475864       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue4 > BusyAgentCount=0
I0327 14:16:04.475868       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue4 > TotalAgentCount=3
I0327 14:16:04.475876       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue4 > BusyAgentPercentage=0
I0327 14:16:04.475882       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > ScheduledJobsCount=0
I0327 14:16:04.475886       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > RunningJobsCount=1
I0327 14:16:04.475890       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > UnfinishedJobsCount=8
I0327 14:16:04.475894       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > WaitingJobsCount=7
I0327 14:16:04.475901       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > IdleAgentCount=0
I0327 14:16:04.475906       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > BusyAgentCount=1
I0327 14:16:04.475910       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > TotalAgentCount=1
I0327 14:16:04.475914       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue5 > BusyAgentPercentage=100
I0327 14:16:04.475926       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue6 > IdleAgentCount=1
I0327 14:16:04.475941       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue6 > BusyAgentCount=0
I0327 14:16:04.475980       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue6 > TotalAgentCount=1
I0327 14:16:04.475993       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue6 > BusyAgentPercentage=0
I0327 14:16:04.475998       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue7 > IdleAgentCount=1
I0327 14:16:04.476002       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue7 > BusyAgentCount=0
I0327 14:16:04.476006       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue7 > TotalAgentCount=1
I0327 14:16:04.476014       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue7 > BusyAgentPercentage=0
I0327 14:16:04.476026       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue8 > IdleAgentCount=5
I0327 14:16:04.476037       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue8 > BusyAgentCount=0
I0327 14:16:04.476050       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue8 > TotalAgentCount=5
I0327 14:16:04.476054       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue8 > BusyAgentPercentage=0
I0327 14:16:04.476061       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > UnfinishedJobsCount=1
I0327 14:16:04.476065       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > WaitingJobsCount=0
I0327 14:16:04.476069       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > IdleAgentCount=4
I0327 14:16:04.476074       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > BusyAgentCount=1
I0327 14:16:04.476078       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > TotalAgentCount=5
I0327 14:16:04.476084       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > BusyAgentPercentage=20
I0327 14:16:04.476091       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > ScheduledJobsCount=0
I0327 14:16:04.476095       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue9 > RunningJobsCount=1
I0327 14:16:04.476099       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > ScheduledJobsCount=2
I0327 14:16:04.476103       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > RunningJobsCount=1
I0327 14:16:04.476110       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > UnfinishedJobsCount=6
I0327 14:16:04.476114       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > WaitingJobsCount=3
I0327 14:16:04.476118       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > IdleAgentCount=0
I0327 14:16:04.476122       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > BusyAgentCount=1
I0327 14:16:04.476127       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > TotalAgentCount=1
I0327 14:16:04.476132       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue10 > BusyAgentPercentage=100
I0327 14:16:04.476140       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue11 > IdleAgentCount=1
I0327 14:16:04.476144       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue11 > BusyAgentCount=0
I0327 14:16:04.476149       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue11 > TotalAgentCount=1
I0327 14:16:04.476154       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue11 > BusyAgentPercentage=0
I0327 14:16:04.476159       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue12 > IdleAgentCount=1
I0327 14:16:04.476168       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue12 > BusyAgentCount=0
I0327 14:16:04.476172       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue12 > TotalAgentCount=1
I0327 14:16:04.476177       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue12 > BusyAgentPercentage=0
I0327 14:16:04.476182       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue13 > IdleAgentCount=1
I0327 14:16:04.476187       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue13 > BusyAgentCount=0
I0327 14:16:04.476196       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue13 > TotalAgentCount=1
I0327 14:16:04.476205       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue13 > BusyAgentPercentage=0
I0327 14:16:04.476210       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue14 > IdleAgentCount=2
I0327 14:16:04.476214       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue14 > BusyAgentCount=0
I0327 14:16:04.476219       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue14 > TotalAgentCount=2
I0327 14:16:04.476226       1 buildkite.go:363] Buildkite > Org=MYORG > Queue=queue14 > BusyAgentPercentage=0
I0327 14:16:04.476288       1 storage.go:46] metric buildkite_total_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476326       1 storage.go:46] metric buildkite_total_scheduled_jobs_count successfully scraped and stored.
I0327 14:16:04.476347       1 storage.go:46] metric buildkite_total_running_jobs_count successfully scraped and stored.
I0327 14:16:04.476367       1 storage.go:46] metric buildkite_total_unfinished_jobs_count successfully scraped and stored.
I0327 14:16:04.476399       1 storage.go:46] metric buildkite_total_waiting_jobs_count successfully scraped and stored.
I0327 14:16:04.476411       1 storage.go:46] metric buildkite_total_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476421       1 storage.go:46] metric buildkite_total_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476434       1 storage.go:46] metric buildkite_total_total_agent_count successfully scraped and stored.
I0327 14:16:04.476447       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476468       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476482       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476491       1 storage.go:46] metric buildkite_scheduled_jobs_count successfully scraped and stored.
I0327 14:16:04.476504       1 storage.go:46] metric buildkite_running_jobs_count successfully scraped and stored.
I0327 14:16:04.476515       1 storage.go:46] metric buildkite_unfinished_jobs_count successfully scraped and stored.
I0327 14:16:04.476533       1 storage.go:46] metric buildkite_waiting_jobs_count successfully scraped and stored.
I0327 14:16:04.476542       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476551       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476556       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476580       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476585       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476595       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476600       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476613       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.476617       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476632       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476637       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476653       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476661       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476670       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476675       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476687       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.476691       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476705       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.476712       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476725       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476729       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476739       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476743       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476756       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476760       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476770       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476774       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476787       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476793       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476806       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476829       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476842       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.476846       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476856       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476864       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476877       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476881       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.476894       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.476900       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.476911       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476930       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476940       1 storage.go:41] metric buildkite_waiting_jobs_count already has value, overwriting...
I0327 14:16:04.476947       1 storage.go:46] metric buildkite_waiting_jobs_count successfully scraped and stored.
I0327 14:16:04.476957       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.476965       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.476976       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.476980       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.476990       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.476998       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.477023       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.477027       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.477035       1 storage.go:41] metric buildkite_scheduled_jobs_count already has value, overwriting...
I0327 14:16:04.477042       1 storage.go:46] metric buildkite_scheduled_jobs_count successfully scraped and stored.
I0327 14:16:04.477062       1 storage.go:41] metric buildkite_running_jobs_count already has value, overwriting...
I0327 14:16:04.477066       1 storage.go:46] metric buildkite_running_jobs_count successfully scraped and stored.
I0327 14:16:04.477091       1 storage.go:41] metric buildkite_unfinished_jobs_count already has value, overwriting...
I0327 14:16:04.477118       1 storage.go:46] metric buildkite_unfinished_jobs_count successfully scraped and stored.
I0327 14:16:04.477162       1 storage.go:41] metric buildkite_waiting_jobs_count already has value, overwriting...
I0327 14:16:04.477194       1 storage.go:46] metric buildkite_waiting_jobs_count successfully scraped and stored.
I0327 14:16:04.477710       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.477728       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.477740       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.477744       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.477754       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.477758       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.477768       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.477781       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.477791       1 storage.go:41] metric buildkite_scheduled_jobs_count already has value, overwriting...
I0327 14:16:04.477806       1 storage.go:46] metric buildkite_scheduled_jobs_count successfully scraped and stored.
I0327 14:16:04.477816       1 storage.go:41] metric buildkite_running_jobs_count already has value, overwriting...
I0327 14:16:04.477820       1 storage.go:46] metric buildkite_running_jobs_count successfully scraped and stored.
I0327 14:16:04.477830       1 storage.go:41] metric buildkite_unfinished_jobs_count already has value, overwriting...
I0327 14:16:04.477834       1 storage.go:46] metric buildkite_unfinished_jobs_count successfully scraped and stored.
I0327 14:16:04.477843       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.477866       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.477894       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.477898       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.477908       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.477928       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.477966       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.477985       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.478012       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.478047       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.478083       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.478109       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.478137       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.478195       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.478276       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.478353       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.478409       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.478436       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.478498       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.478583       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.478606       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.478672       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.478694       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.478744       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.478768       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.478773       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.478782       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.478787       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.478841       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.478855       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.478863       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.478865       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.478876       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.478880       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.478938       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.478954       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.478964       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.478978       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.478988       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.478992       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.479005       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0327 14:16:04.479010       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0327 14:16:04.479049       1 storage.go:41] metric buildkite_busy_agent_count already has value, overwriting...
I0327 14:16:04.479064       1 storage.go:46] metric buildkite_busy_agent_count successfully scraped and stored.
I0327 14:16:04.479073       1 storage.go:41] metric buildkite_total_agent_count already has value, overwriting...
I0327 14:16:04.479076       1 storage.go:46] metric buildkite_total_agent_count successfully scraped and stored.
I0327 14:16:04.479085       1 storage.go:41] metric buildkite_busy_agent_percentage already has value, overwriting...
I0327 14:16:04.479098       1 storage.go:46] metric buildkite_busy_agent_percentage successfully scraped and stored.
I0327 14:16:04.540840       1 shared_informer.go:270] caches populated
I0327 14:16:04.540859       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0327 14:16:04.540886       1 shared_informer.go:270] caches populated
I0327 14:16:04.540890       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0327 14:16:04.540942       1 shared_informer.go:270] caches populated
I0327 14:16:04.540962       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0327 14:16:04.541222       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"399c129c-d96a-4738-b3eb-3ffce51867e8\" [] issuer=\"<self>\" (2022-07-16 06:16:43 +0000 UTC to 2052-07-08 07:16:43 +0000 UTC (now=2023-03-27 14:16:04.541194292 +0000 UTC))"
I0327 14:16:04.541465       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" certDetail="\"localhost@1679926564\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1679926564\" (2023-03-27 13:16:03 +0000 UTC to 2024-03-26 13:16:03 +0000 UTC (now=2023-03-27 14:16:04.541439762 +0000 UTC))"
I0327 14:16:04.541677       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1679926564\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1679926564\" (2023-03-27 13:16:04 +0000 UTC to 2024-03-26 13:16:04 +0000 UTC (now=2023-03-27 14:16:04.541654672 +0000 UTC))"
I0327 14:16:04.541801       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"399c129c-d96a-4738-b3eb-3ffce51867e8\" [] issuer=\"<self>\" (2022-07-16 06:16:43 +0000 UTC to 2052-07-08 07:16:43 +0000 UTC (now=2023-03-27 14:16:04.541779702 +0000 UTC))"
I0327 14:16:04.541834       1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"4074b60f-256f-42b1-b7d5-144c666f2fd2\" [] issuer=\"<self>\" (2022-07-16 06:18:27 +0000 UTC to 2052-07-08 07:18:27 +0000 UTC (now=2023-03-27 14:16:04.541814712 +0000 UTC))"
I0327 14:16:04.542023       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" certDetail="\"localhost@1679926564\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1679926564\" (2023-03-27 13:16:03 +0000 UTC to 2024-03-26 13:16:03 +0000 UTC (now=2023-03-27 14:16:04.542001902 +0000 UTC))"
I0327 14:16:04.542208       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1679926564\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1679926564\" (2023-03-27 13:16:04 +0000 UTC to 2024-03-26 13:16:04 +0000 UTC (now=2023-03-27 14:16:04.542188022 +0000 UTC))"
I0327 14:16:05.587011       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:05.591300       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787 200 OK in 4 milliseconds
I0327 14:16:05.593336       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798&timeout=5m19s&timeoutSeconds=319&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:05.593385       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:05.596523       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:05.598435       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787 200 OK in 1 milliseconds
I0327 14:16:05.600350       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798&timeout=5m29s&timeoutSeconds=329&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:05.600401       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:05.797848       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:05.800195       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160787 200 OK in 2 milliseconds
I0327 14:16:05.801979       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798&timeout=7m34s&timeoutSeconds=454&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:05.802058       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:07.444632       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:07.449639       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798 200 OK in 4 milliseconds
I0327 14:16:07.451432       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160813&timeout=6m4s&timeoutSeconds=364&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:07.451498       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:07.682790       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:07.685391       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798 200 OK in 2 milliseconds
I0327 14:16:07.686828       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160813&timeout=6m35s&timeoutSeconds=395&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:07.686872       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)
I0327 14:16:08.703971       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 14:16:08.706861       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160798 200 OK in 2 milliseconds
I0327 14:16:08.708741       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160827&timeout=6m24s&timeoutSeconds=384&watch=true 403 Forbidden in 1 milliseconds
E0327 14:16:08.708780       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: unknown (get configmaps)

Tomoyuki Sugiyama · Answer 3 · Tue Mar 28 2023 08:09:15 GMT+0800 (China Standard Time)

Fixed Failed to watch *v1.ConfigMap: unknown (get configmaps) error
when I append "watch" variable to buildscaler-resource-reader in rbac.yaml

But I cant get expected result with kubectl describe apiservices.apiregistration.k8s.io v1beta1.external.metrics.k8s.io

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: buildscaler-resource-reader
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  - services
  - configmaps
  verbs:
  - get
  - list
  - watch

Logs

I0327 23:53:11.088996       1 main.go:95] using buildkite scraper & metrics provider
I0327 23:53:11.089157       1 buildkite.go:198] Collecting agent metrics for all queues
I0327 23:53:11.264087       1 serving.go:341] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0327 23:53:11.264537       1 dynamic_serving_content.go:110] "Loaded a new cert/key pair" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0327 23:53:11.457531       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 10 milliseconds
I0327 23:53:11.458210       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I0327 23:53:11.459195       1 config.go:679] Not requested to run hook priority-and-fairness-config-consumer
I0327 23:53:11.460642       1 healthz.go:174] Installing health checkers for (/healthz): "ping","log","poststarthook/max-in-flight-filter"
I0327 23:53:11.460803       1 healthz.go:174] Installing health checkers for (/livez): "ping","log","poststarthook/max-in-flight-filter"
I0327 23:53:11.460975       1 healthz.go:174] Installing health checkers for (/readyz): "ping","log","poststarthook/max-in-flight-filter","shutdown"
I0327 23:53:11.461212       1 genericapiserver.go:419] [graceful-termination] ShutdownTimeout=1m0s
I0327 23:53:11.467382       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 5 milliseconds
I0327 23:53:11.468026       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" certDetail="\"localhost@1679961191\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1679961191\" (2023-03-27 22:53:11 +0000 UTC to 2024-03-26 22:53:11 +0000 UTC (now=2023-03-27 23:53:11.467997852 +0000 UTC))"
I0327 23:53:11.468223       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1679961191\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1679961191\" (2023-03-27 22:53:11 +0000 UTC to 2024-03-26 22:53:11 +0000 UTC (now=2023-03-27 23:53:11.468192452 +0000 UTC))"
I0327 23:53:11.468295       1 secure_serving.go:266] Serving securely on [::]:6443
I0327 23:53:11.468319       1 genericapiserver.go:378] [graceful-termination] waiting for shutdown to be initiated
I0327 23:53:11.468348       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0327 23:53:11.468368       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0327 23:53:11.468415       1 dynamic_serving_content.go:129] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0327 23:53:11.468491       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0327 23:53:11.468650       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0327 23:53:11.468670       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0327 23:53:11.468692       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0327 23:53:11.468697       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0327 23:53:11.468779       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.468805       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.469369       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.469449       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.469696       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.469708       1 reflector.go:255] Listing and watching *v1.ConfigMap from pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167
I0327 23:53:11.471298       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 2 milliseconds
I0327 23:53:11.471646       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0327 23:53:11.472166       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 200 OK in 2 milliseconds
I0327 23:53:11.473619       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201501966&timeout=8m1s&timeoutSeconds=481&watch=true 200 OK in 1 milliseconds
I0327 23:53:11.473758       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201501966&timeout=9m42s&timeoutSeconds=582&watch=true 200 OK in 1 milliseconds
I0327 23:53:11.473996       1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201501966&timeout=8m19s&timeoutSeconds=499&watch=true 200 OK in 1 milliseconds
I0327 23:53:11.568690       1 shared_informer.go:270] caches populated
I0327 23:53:11.568760       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0327 23:53:11.568718       1 shared_informer.go:270] caches populated
I0327 23:53:11.568817       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0327 23:53:11.568733       1 shared_informer.go:270] caches populated
I0327 23:53:11.568862       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0327 23:53:11.569225       1 tlsconfig.go:178] "Loaded client CA" index=0 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"399c129c-d96a-4738-b3eb-3ffce51867e8\" [] issuer=\"<self>\" (2022-07-16 06:16:43 +0000 UTC to 2052-07-08 07:16:43 +0000 UTC (now=2023-03-27 23:53:11.569197705 +0000 UTC))"
I0327 23:53:11.569279       1 tlsconfig.go:178] "Loaded client CA" index=1 certName="client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" certDetail="\"4074b60f-256f-42b1-b7d5-144c666f2fd2\" [] issuer=\"<self>\" (2022-07-16 06:18:27 +0000 UTC to 2052-07-08 07:18:27 +0000 UTC (now=2023-03-27 23:53:11.569245374 +0000 UTC))"
I0327 23:53:11.569507       1 tlsconfig.go:200] "Loaded serving cert" certName="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key" certDetail="\"localhost@1679961191\" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer=\"localhost-ca@1679961191\" (2023-03-27 22:53:11 +0000 UTC to 2024-03-26 22:53:11 +0000 UTC (now=2023-03-27 23:53:11.569486295 +0000 UTC))"
I0327 23:53:11.569729       1 named_certificates.go:53] "Loaded SNI cert" index=0 certName="self-signed loopback" certDetail="\"apiserver-loopback-client@1679961191\" [serving] validServingFor=[apiserver-loopback-client] issuer=\"apiserver-loopback-client-ca@1679961191\" (2023-03-27 22:53:11 +0000 UTC to 2024-03-26 22:53:11 +0000 UTC (now=2023-03-27 23:53:11.569710455 +0000 UTC))"

Tomoyuki Sugiyama · Answer 4 · Tue Mar 28 2023 09:50:46 GMT+0800 (China Standard Time)

I have tried to check a buildscaler-apiserver is runnning or not by using the ubuntu container and the buildscaler-apiserver seems to be working.

Logs

curl from ubuntu container in the same cluster

# curl -k 'https://10.91.15.221:6443/apis/external.metrics.k8s.io/v1beta1'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/apis/external.metrics.k8s.io/v1beta1\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

buildscaler-apiserver responsed

I0328 01:41:49.038110       1 storage.go:41] metric buildkite_idle_agent_count already has value, overwriting...
I0328 01:41:49.038114       1 storage.go:46] metric buildkite_idle_agent_count successfully scraped and stored.
I0328 01:41:49.898785       1 round_trippers.go:454] POST https://10.91.128.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s 201 Created in 5 milliseconds
I0328 01:41:49.900117       1 authorization.go:73] "Forbidden" URI="/apis/external.metrics.k8s.io/v1beta1" Reason=""
I0328 01:41:49.900603       1 httplog.go:104] "HTTP" verb="GET" URI="/apis/external.metrics.k8s.io/v1beta1" latency="7.73579ms" userAgent="curl/7.81.0" audit-ID="e120792e-b888-43f1-9ca3-80eaf916bb3c" srcIP="10.91.192.37:44004" resp=403

Paweł Bojanowski · Answer 5 · Tue Mar 28 2023 18:11:21 GMT+0800 (China Standard Time)

I0327 14:16:08.708741 1 round_trippers.go:454] GET https://10.91.128.1:443/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=201160827&timeout=6m24s&timeoutSeconds=384&watch=true 403 Forbidden in 1 milliseconds

suggests that something might be wrong with RBAC. Did you deployed using deploy/deploy.sh script? There are several Roles, ClusterRoles and ServiceAccounts in the deploy/rbac.yaml and deploy/rbac-kube-system.yaml files that need to be present in the cluster and buildscaler-apiserver.

Tomoyuki Sugiyama · Answer 6 · Tue Mar 28 2023 18:55:58 GMT+0800 (China Standard Time)

yes, I use deploy script.

% ./deploy.sh "$NAMESPACE"                                                                                                  (git)-[tags/v2.3.1]
NAME              TYPE     DATA   AGE
buildkite-agent   Opaque   1      5d6h
rolebinding.rbac.authorization.k8s.io/buildscaler-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/buildscaler-hpa-controller created
rolebinding.rbac.authorization.k8s.io/buildscaler-auth-reader configured
serviceaccount/buildscaler-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/buildkite:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/buildscaler-resource-reader created
clusterrole.rbac.authorization.k8s.io/buildscaler-server-resources created
clusterrole.rbac.authorization.k8s.io/buildscaler-resource-reader created
service/buildscaler-apiserver created
deployment.apps/buildscaler-apiserver created
apiservice.apiregistration.k8s.io/v1beta1.external.metrics.k8s.io created

Script

% cat deploy.sh                                                                                                             (git)-[tags/v2.3.1]
#!/bin/sh

set -e

readonly namespace=$1

if [ -z "$namespace" ]
then
    echo "usage: $0 <namespace>"
    exit 1
fi

kubectl --namespace "$namespace" get secret buildkite-agent || {
    echo "secret/buildkite-agent not found in namespace $namespace"
    echo "To create it: kubectl --namespace=$namespace create secret generic buildkite-agent --from-literal=token=<Buildkite Agent Token>"
    exit 1
}

kubectl --namespace kube-system apply -f rbac-kube-system.yaml

kubectl_apply() {
    kubectl --namespace "$namespace" apply -f "$*"
}
sed "s/##NAMESPACE##/$namespace/" < rbac.yaml | kubectl_apply -
kubectl_apply service.yaml
kubectl_apply buildkite-deployment.yaml
sed "s/##NAMESPACE##/$namespace/" < apiservice.yaml | kubectl_apply -

Tomoyuki Sugiyama · Answer 7 · Tue Mar 28 2023 19:21:04 GMT+0800 (China Standard Time)

My gke cluster is enabled Workload Identity, is it no problem?

Paweł Bojanowski · Answer 8 · Tue Mar 28 2023 19:32:33 GMT+0800 (China Standard Time)

My gke cluster is enabled Workload Identity, is it no problem?

Yes, I think this is a root cause of an issue. I think you need to create IAM role and then annotate buildscaler-apiserver service account.
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to
I think assigning this role is a safe choice: https://cloud.google.com/iam/docs/understanding-roles#container.developer
if you need to restrict access further, you may check deploy/rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: buildscaler-server-resources
rules:
- apiGroups:
  - external.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: buildscaler-resource-reader
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  - services
  - configmaps
  verbs:
  - get
  - list

buildscaler needs write permission for external.metrics.k8s.io and the read access to the ones listed in buildscaler-resource-reader roles

Tomoyuki Sugiyama · Answer 9 · Tue Mar 28 2023 20:26:16 GMT+0800 (China Standard Time)

Yes, I think this is a root cause of an issue. I think you need to create IAM role and then annotate buildscaler-apiserver service account.

I tried this, but I cant get expected behavior.

Message:               failing or missing response from https://10.91.192.45:6443/apis/external.metrics.k8s.io/v1beta1: Get "https://10.91.192.45:6443/apis/external.metrics.k8s.io/v1beta1": context deadline exceeded

Create gcp service account with these IAM roles.

gcloud iam service-accounts add-iam-policy-binding --role \
  roles/iam.workloadIdentityUser --member \
  "serviceAccount:ubd-dev.svc.id.goog[buildkite/buildscaler-apiserver]" \
  dev-buildscaler-apiserver@my-project.iam.gserviceaccount.com

gcloud projects add-iam-policy-binding my-project \
    --member "serviceAccount:dev-buildscaler-apiserver@my-project.iam.gserviceaccount.com" \
    --role "roles/container.developer"

and set annotation.

kind: ServiceAccount
apiVersion: v1
metadata:
  name: buildscaler-apiserver
  annotations:
    iam.gke.io/gcp-service-account: dev-buildscaler-apiserver@my-project.iam.gserviceaccount.com

Tomoyuki Sugiyama · Answer 10 · Wed May 10 2023 18:37:12 GMT+0800 (China Standard Time)

Hi @hidalgopl, thanks for your helpful and I solved this problem.

It was our network problem, sorry.
I needed to set firewall rule.

gcloud compute firewall-rules create allow-prometheus-adapter \
    --action ALLOW \
    --direction INGRESS \
    --source-ranges 172.16.0.0/28 \ # Master CIDR Block
    --rules tcp:6443 \
    --network {{VPC Network Name}}

ref from: kubernetes-sigs/prometheus-adapter#134

result:
kubectl describe apiservices.apiregistration.k8s.io v1beta1.external.metrics.k8s.io

Status:
  Conditions:
    Last Transition Time:  2023-05-10T10:19:14Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

Thank you so much.
I will close this issue.