Namespace deletion is stuck when using namespace selector startup mode
nicolasadeo opened this issue · comments
Long story short
When running a kopf operator with namespace selector AND deleting this namespace that contains a resource with a finalizer: kopf may not call the delete handler method.
Deleting the namespace stops the watch-stream for this namespace before the resources contained in that namespace get the chance have its delete method to be called.
Therefore, the finalizer is never removed and the namespace deletion is stuck waiting all resources to be deleted (but they won't since the watch-stream is stopped).
We manage to find a workaround using the clusterwide setting.
Earlier, we say "may not call"; in some cases the delete handler is called but it is quickly canceled by a CancelledError coming from kopf. We suppose it may be relatee to the event queue size. In the simple example provided, we never manage to get the delete handler to be called.
Kopf version
1.36.2
Kubernetes version
1.27.4
Python version
3.11.1
Code
import kopf
import asyncio
import threading
import logging
from kopf._core.actions import loggers
@kopf.on.create('kopfexamples')
def create_fn(**kwargs):
print('CREATED!!!')
@kopf.on.delete('kopfexamples')
async def delete_fn(**kwargs):
await asyncio.sleep(2)
print(f'DELETED!!!')
@kopf.on.startup() # type: ignore
def configure(settings: kopf.OperatorSettings, **_) -> None:
settings.posting.level = logging.DEBUG
loggers.configure(
debug=True,
verbose=False,
quiet=False,
log_format=loggers.LogFormat.PLAIN, # loggers.LogFormat.JSON,
log_refkey="k8s_obj",
log_prefix=False,
)
def kopf_thread():
asyncio.run(kopf.operator(namespaces=["example"]))
def main():
logging.getLogger("kopf").setLevel(logging.DEBUG)
thread = threading.Thread(target=kopf_thread)
thread.start()
thread.join()
if __name__ == "__main__":
main()
Logs
Adding the finalizer, thus preventing the actual deletion.
Patching with: {'metadata': {'finalizers': ['kopf.zalando.org/KopfFinalizerMarker']}}
Creation is in progress: {'apiVersion': 'kopf.dev/v1', 'kind': 'KopfExample', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"kopf.dev/v1","kind":"KopfExample","metadata":{"annotations":{"someannotation":"somevalue"},"labels":{"somelabel":"somevalue"},"name":"kopf-example-1","namespace":"example"},"spec":{"duration":"1m","field":"value","items":["item1","item2"]}}\n', 'someannotation': 'somevalue'}, 'creationTimestamp': '2024-01-10T15:07:37Z', 'finalizers': ['kopf.zalando.org/KopfFinalizerMarker'], 'generation': 1, 'labels': {'somelabel': 'somevalue'}, 'managedFields': [{'apiVersion': 'kopf.dev/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"kopf.zalando.org/KopfFinalizerMarker"': {}}}}, 'manager': 'kopf', 'operation': 'Update', 'time': '2024-01-10T15:07:37Z'}, {'apiVersion': 'kopf.dev/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kubectl.kubernetes.io/last-applied-configuration': {}, 'f:someannotation': {}}, 'f:labels': {'.': {}, 'f:somelabel': {}}}, 'f:spec': {'.': {}, 'f:duration': {}, 'f:field': {}, 'f:items': {}}}, 'manager': 'kubectl-client-side-apply', 'operation': 'Update', 'time': '2024-01-10T15:07:37Z'}], 'name': 'kopf-example-1', 'namespace': 'example', 'resourceVersion': '334932', 'uid': '829ca9ed-c7a5-4426-baab-7b3762f9b7fb'}, 'spec': {'duration': '1m', 'field': 'value', 'items': ['item1', 'item2']}}
Handler 'create_fn' is invoked.
Handler 'create_fn' succeeded.
Creation is processed: 1 succeeded; 0 failed.
Patching with: {'metadata': {'annotations': {'kopf.zalando.org/last-handled-configuration': '{"spec":{"duration":"1m","field":"value","items":["item1","item2"]},"metadata":{"labels":{"somelabel":"somevalue"},"annotations":{"someannotation":"somevalue"}}}\n'}}}
CREATED!!!
Something has changed, but we are not interested (the essence is the same).
Handling cycle is finished, waiting for new changes.
Stopping the watch-stream for kopfexamples.v1.kopf.dev in 'example'.
Additional information
After some code digging trying to understand what's going on, we think the problem is here and the namespace is removed from the watching list without any check:
kopf/kopf/_core/reactor/observation.py
Line 202 in 91e8fd6
more code snippets.
The CRD we use:
# A demo CRD for the Kopf example operators.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: kopfexamples.kopf.dev
spec:
scope: Namespaced
group: kopf.dev
names:
kind: KopfExample
plural: kopfexamples
singular: kopfexample
shortNames:
- kopfexes
- kopfex
- kexes
- kex
versions:
- name: v1
served: true
storage: true
subresources: { status: { } } # comment/uncomment for experiments
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
x-kubernetes-preserve-unknown-fields: true
status:
type: object
x-kubernetes-preserve-unknown-fields: true
additionalPrinterColumns:
- name: Duration
type: string
priority: 0
jsonPath: .spec.duration
description: For how long the pod should sleep.
- name: Children
type: string
priority: 0
jsonPath: .status.create_fn.children
description: The children pods created.
- name: Message
type: string
priority: 0
jsonPath: .status.create_fn.message
description: As returned from the handler (sometimes).
and the resource manifest:
apiVersion: kopf.dev/v1
kind: KopfExample
metadata:
name: kopf-example-1
namespace: example
labels:
somelabel: somevalue
annotations:
someannotation: somevalue
spec:
duration: 1m
field: value
items:
- item1
- item2