Giters

nolar / kopf

A Python framework to write Kubernetes operators in just a few lines of code

Home Page:https://kopf.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Failure to shutdown when ran out of valid credentials for authentication (stop_flag appears to be problem).

GrahamDumpleton opened this issue · comments

commented

Long story short

The short of it is that when authentication fails, possibly a transient issue (currently original trigger as to why is unknown), the kopf main event loop is shutting down but not fully exiting. The result is the operator is being left in a dead state, but pod doesn't exit on its own.

Of note:

  • Not using kopf run command line client, instead was coding kopf startup explicitly.
  • Not using normal in-cluster REST API service account token. Instead using a secret of type kubernetes.io/service-account-token with annotation kubernetes.io/service-account.name to link it to the service account that operator runs as. This was done with intent to avoid any possible issues with token refreshing in latest Kubernetes version.

Also of note:

  • Providing a health check via a liveness probe acts as fails safe to restart pod.
  • Using kopf.on.login with kopf.login_via_pykube doesn't seem to help in as much as doesn't result in retries in way replicating the issue.

After investigation, the issue seems to be related to stop_flag being supplied as an argument.

If comment out that argument:

            _event_loop.run_until_complete(
                kopf.operator(
                    clusterwide=True,
                    # stop_flag=_stop_flag,
                    # liveness_endpoint="http://0.0.0.0:8080/healthz",
                )
            )

then it does exit as expected.

Instead of doing that, can leave stop_flag argument in place and instead use:

@kopf.on.cleanup()
async def cleanup_fn(logger, **kwargs):
    logger.info("Stopping kopf framework main loop.")

    # Workaround for possible kopf bug, set stop flag.

    _stop_flag.set()

to explicitly set the event for stopping the loop. Things then shutdown as expect when authentication fails.

Kopf version

kopf[full-auth]==1.35.5

Kubernetes version

1.23.4

Python version

3.10

Code

# Embedded kopf main loop code after tried various things to replicate.

@kopf.on.startup()
def configure(settings: kopf.OperatorSettings, **_):
    settings.posting.level = logging.DEBUG


@kopf.on.login()
def login_fn(**kwargs):
    return kopf.login_via_pykube(**kwargs)


@kopf.on.cleanup()
async def cleanup_fn(logger, **kwargs):
    logger.info("Stopping kopf framework main loop.")


_stop_flag = Event()


def run_kopf():
    """Run kopf in a separate thread and wait for it to complete."""

    def worker():
        logger.info("Starting kopf framework main loop.")

        # Need to create and set the event loop since this isn't being
        # called in the main thread.

        global _event_loop  # pylint: disable=invalid-name,global-statement

        _event_loop = asyncio.new_event_loop()
        asyncio.set_event_loop(_event_loop)

        # Schedule background task to purge namespaces.

        _event_loop.create_task(daemons.purge_namespaces())

        with contextlib.closing(_event_loop):
            # Run event loop until flagged to shutdown.

            _event_loop.run_until_complete(
                kopf.operator(
                    clusterwide=True,
                    stop_flag=_stop_flag,
                    # liveness_endpoint="http://0.0.0.0:8080/healthz",
                )
            )

            logger.info("Closing asyncio event loop.")

        logger.info("Exiting kopf framework main loop.")

    thread = Thread(target=worker)

    # Startup kopf framework.

    thread.start()

    return thread


def shutdown(signum, frame):
    logger.info("Signal handler called with signal %s.", signum)
    if _event_loop:
        _stop_flag.set()


if __name__ == "__main__":
    signal.signal(signal.SIGINT, shutdown)
    signal.signal(signal.SIGTERM, shutdown)

    thread = run_kopf()
    thread.join()

# Details of Kubernetes secret. Magic comments relate to ytt templating.

apiVersion: v1
kind: Secret
metadata:
  name: secrets-manager-token
  namespace: #@ data.values.operator.namespace
  annotations:
    kubernetes.io/service-account.name: "secrets-manager"
    kapp.k14s.io/change-rule: #@ "upsert after upserting secrets.{}/service-accounts".format(data.values.operator.apiGroup)
type: kubernetes.io/service-account-token

# Details of service account. Must exist before creating secret.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secrets-manager
  namespace: #@ data.values.operator.namespace
  annotations:
    kapp.k14s.io/change-group: #@ "secrets.{}/service-accounts".format(data.values.operator.apiGroup)
    #! Following currently needed for kapp on OpenShift.
    kapp.k14s.io/create-strategy: fallback-on-update
    kapp.k14s.io/update-strategy: skip

# Details from deployment of operator.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secrets-manager
  namespace: #@ data.values.operator.namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      deployment: secrets-manager
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        deployment: secrets-manager
    spec:
      serviceAccountName: secrets-manager
      automountServiceAccountToken: false
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        runAsNonRoot: true
        runAsUser: 1001
        #! seccompProfile:
        #!   type: RuntimeDefault
      containers:
      - name: operator
        #@ image = image_reference("secrets-manager")
        image: #@ image
        imagePullPolicy: #@ image_pull_policy(image)
        #! livenessProbe:
        #!   httpGet:
        #!     path: /healthz
        #!     port: 8080
        volumeMounts:
        - name: config
          mountPath: /opt/app-root/config/
        - name: token
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
      volumes:
      - name: config
        secret:
          secretName: #@ "{}-config".format(data.values.operator.namePrefix)
      - name: token
        secret:
          secretName: secrets-manager-token

Logs

INFO:kopf.objects:Processing rule 0 from secretcopier/xxx-ingress-secrets against namespace xxx.
INFO:kopf.objects:Timer 'secretcopier_reconcile' succeeded.
DEBUG:kopf.objects:Timer 'secretinjector_reconcile' is invoked.
DEBUG:kopf.objects:Timer 'secretcopier_reconcile' is invoked.
INFO:kopf.objects:Timer 'secretinjector_reconcile' succeeded.
INFO:kopf.objects:Processing rule 0 from secretcopier/xxx-ingress-secrets against namespace xxx.
INFO:kopf.objects:Timer 'secretcopier_reconcile' succeeded.
DEBUG:kopf.objects:Handler 'secret_event' is invoked.
INFO:kopf.objects:Handler 'secret_event' succeeded.
DEBUG:kopf.objects:Timer 'secretinjector_reconcile' is invoked.
DEBUG:kopf.objects:Timer 'secretcopier_reconcile' is invoked.
INFO:kopf._core.engines.activities:Re-authentication has been initiated.
DEBUG:kopf.activities.authentication:Activity 'login_fn' is invoked.
DEBUG:kopf.activities.authentication:Pykube is configured in cluster with service account.
INFO:kopf.activities.authentication:Activity 'login_fn' succeeded.
INFO:kopf._core.engines.activities:Re-authentication has finished.
ERROR:kopf._core.reactor.running:Poster of events has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://10.96.0.1:443/api/v1/namespaces/xxx/events')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIUnauthorizedError: ('Unauthorized', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Unauthorized', 'reason': 'Unauthorized', 'code': 401})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/events.py", line 76, in post_event
    await api.post(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 133, in post
    response = await request(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 47, in wrapper
    await vault.invalidate(key, exc=e)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/structs/credentials.py", line 261, in invalidate
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
DEBUG:kopf._core.reactor.running:Admission validating configuration manager is cancelled.
DEBUG:kopf._core.reactor.running:Resource observer is cancelled.
DEBUG:kopf._core.reactor.running:Admission mutating configuration manager is cancelled.
DEBUG:kopf._core.reactor.running:Admission insights chain is cancelled.
DEBUG:kopf._core.reactor.running:Namespace observer is cancelled.
DEBUG:kopf._core.reactor.running:Credentials retriever is cancelled.
DEBUG:kopf._core.reactor.running:Admission webhook server is cancelled.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretimporters.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for serviceaccounts.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secrets.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for namespaces.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretexporters.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretinjectors.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretcopiers.v1beta1.secrets.xxx.dev cluster-wide.
WARNING:kopf.objects:Timer 'secretcopier_reconcile' did not exit in time. Leaving it orphaned.
WARNING:kopf.objects:Timer 'secretinjector_reconcile' did not exit in time. Leaving it orphaned.
DEBUG:kopf._core.reactor.orchestration:Streaming tasks are stopped: finishing normally; tasks left: set()
DEBUG:kopf._core.reactor.running:Multidimensional multitasker is cancelled.
DEBUG:kopf._core.reactor.running:Daemon killer is cancelled.
ERROR:kopf.objects:Timer 'secretcopier_reconcile' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/app-root/src/handlers/secretcopier.py", line 21, in secretcopier_reconcile
    reconcile_config(name, body)
  File "/opt/app-root/src/handlers/secretcopier_funcs.py", line 222, in reconcile_config
    for namespace_item in namespace_query:
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 195, in __iter__
    return iter(self.query_cache["objects"])
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 185, in query_cache
    cache["response"] = self.execute().json()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 160, in execute
    r.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://10.96.0.1:443/api/v1/namespaces
DEBUG:kopf.activities.cleanup:Activity 'cleanup_fn' is invoked.
INFO:kopf.activities.cleanup:Stopping kopf framework main loop.
INFO:kopf.activities.cleanup:Activity 'cleanup_fn' succeeded.
DEBUG:kopf._core.reactor.running:Root tasks are stopped: finishing normally; tasks left: set()
ERROR:kopf.objects:Timer 'secretinjector_reconcile' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/app-root/src/handlers/secretinjector.py", line 23, in secretinjector_reconcile
    reconcile_config(name, body)
  File "/opt/app-root/src/handlers/secretinjector_funcs.py", line 258, in reconcile_config
    for namespace_item in namespace_query:
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 195, in __iter__
    return iter(self.query_cache["objects"])
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 185, in query_cache
    cache["response"] = self.execute().json()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 160, in execute
    r.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://10.96.0.1:443/api/v1/namespaces
DEBUG:kopf._core.reactor.running:Hung tasks are stopped: finishing normally; tasks left: set()
Exception in thread Thread-1 (worker):
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://10.96.0.1:443/api/v1/namespaces/xxx/events')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIUnauthorizedError: ('Unauthorized', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Unauthorized', 'reason': 'Unauthorized', 'code': 401})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/app-root/src/main.py", line 58, in worker
    _event_loop.run_until_complete(
  File "/usr/lib64/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/reactor/running.py", line 135, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/reactor/running.py", line 416, in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 238, in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/events.py", line 76, in post_event
    await api.post(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 133, in post
    response = await request(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 47, in wrapper
    await vault.invalidate(key, exc=e)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/structs/credentials.py", line 261, in invalidate
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/

Additional information

It is unknown what the original trigger was for causing authentication to fail, but was assumed to be transient in nature. One can replicate the same condition and see issue with main loop not exiting properly by doing the following.

Using above arrangement of secret of type kubernetes.io/service-account-token with annotation kubernetes.io/service-account.name to link it to the service account that operator runs as, automountServiceAccountToken set to false and with secret instead mounted into the deployment, then delete the secret of type kubernetes.io/service-account-token. This results in token being invalidated and authentication of operator REST API connections being rejected.

At that point you see:

INFO:kopf._core.engines.activities:Re-authentication has been initiated.
DEBUG:kopf.activities.authentication:Activity 'login_fn' is invoked.
DEBUG:kopf.activities.authentication:Pykube is configured in cluster with service account.
INFO:kopf.activities.authentication:Activity 'login_fn' succeeded.
INFO:kopf._core.engines.activities:Re-authentication has finished.
ERROR:kopf._core.reactor.running:Poster of events has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/

with stack traces of:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://10.96.0.1:443/api/v1/namespaces/xxx/events')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIUnauthorizedError: ('Unauthorized', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Unauthorized', 'reason': 'Unauthorized', 'code': 401})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/events.py", line 76, in post_event
    await api.post(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 133, in post
    response = await request(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 47, in wrapper
    await vault.invalidate(key, exc=e)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/structs/credentials.py", line 261, in invalidate
    raise LoginError("Ran out of valid credentials. Consider installing "

This triggered shutdown of kopf.

kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
DEBUG:kopf._core.reactor.running:Admission validating configuration manager is cancelled.
DEBUG:kopf._core.reactor.running:Resource observer is cancelled.
DEBUG:kopf._core.reactor.running:Admission mutating configuration manager is cancelled.
DEBUG:kopf._core.reactor.running:Admission insights chain is cancelled.
DEBUG:kopf._core.reactor.running:Namespace observer is cancelled.
DEBUG:kopf._core.reactor.running:Credentials retriever is cancelled.
DEBUG:kopf._core.reactor.running:Admission webhook server is cancelled.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretimporters.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for serviceaccounts.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secrets.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for namespaces.v1 cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretexporters.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretinjectors.v1beta1.secrets.xxx.dev cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for secretcopiers.v1beta1.secrets.xxx.dev cluster-wide.
WARNING:kopf.objects:Timer 'secretcopier_reconcile' did not exit in time. Leaving it orphaned.
WARNING:kopf.objects:Timer 'secretinjector_reconcile' did not exit in time. Leaving it orphaned.
DEBUG:kopf._core.reactor.orchestration:Streaming tasks are stopped: finishing normally; tasks left: set()
DEBUG:kopf._core.reactor.running:Multidimensional multitasker is cancelled.
DEBUG:kopf._core.reactor.running:Daemon killer is cancelled.
ERROR:kopf.objects:Timer 'secretcopier_reconcile' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/app-root/src/handlers/secretcopier.py", line 21, in secretcopier_reconcile
    reconcile_config(name, body)
  File "/opt/app-root/src/handlers/secretcopier_funcs.py", line 222, in reconcile_config
    for namespace_item in namespace_query:
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 195, in __iter__
    return iter(self.query_cache["objects"])
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 185, in query_cache
    cache["response"] = self.execute().json()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 160, in execute
    r.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://10.96.0.1:443/api/v1/namespaces
DEBUG:kopf.activities.cleanup:Activity 'cleanup_fn' is invoked.
INFO:kopf.activities.cleanup:Stopping kopf framework main loop.
INFO:kopf.activities.cleanup:Activity 'cleanup_fn' succeeded.
DEBUG:kopf._core.reactor.running:Root tasks are stopped: finishing normally; tasks left: set()
ERROR:kopf.objects:Timer 'secretinjector_reconcile' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/app-root/src/handlers/secretinjector.py", line 23, in secretinjector_reconcile
    reconcile_config(name, body)
  File "/opt/app-root/src/handlers/secretinjector_funcs.py", line 258, in reconcile_config
    for namespace_item in namespace_query:
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 195, in __iter__
    return iter(self.query_cache["objects"])
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 185, in query_cache
    cache["response"] = self.execute().json()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/pykube/query.py", line 160, in execute
    r.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://10.96.0.1:443/api/v1/namespaces
DEBUG:kopf._core.reactor.running:Hung tasks are stopped: finishing normally; tasks left: set()
Exception in thread Thread-1 (worker):
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://10.96.0.1:443/api/v1/namespaces/xxx/events')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIUnauthorizedError: ('Unauthorized', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Unauthorized', 'reason': 'Unauthorized', 'code': 401})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/app-root/src/main.py", line 58, in worker
    _event_loop.run_until_complete(
  File "/usr/lib64/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/reactor/running.py", line 135, in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/reactor/running.py", line 416, in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 238, in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_core/engines/posting.py", line 171, in poster
    await events.post_event(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/events.py", line 76, in post_event
    await api.post(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/api.py", line 133, in post
    response = await request(
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 47, in wrapper
    await vault.invalidate(key, exc=e)
  File "/opt/app-root/venv/lib64/python3.10/site-packages/kopf/_cogs/structs/credentials.py", line 261, in invalidate
    raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/

with mix of internal kopf messages and more exceptions as things still tried to run.

The last kopf internal message seemed to be:

DEBUG:kopf._core.reactor.running:Hung tasks are stopped: finishing normally; tasks left: set()

but the run_until_complete() of the event loop never seemed to return.