nolar / kopf

A Python framework to write Kubernetes operators in just a few lines of code

Home Page:https://kopf.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Missing event which comes during processing of the previous event and 2 update handlers

crypo opened this issue · comments

Long story short

An event gets missed in the following scenario:

There are two fields in CRD: fieldA and fieldB
There are two on-update handlers (handler1 and handler2): the first one has no filter, the second one has a filter by fieldB
The first handler takes 1 minute to execute
Create resource.
Submit update for fieldB , then in a few seconds submit a separate update for fieldA

Expected behavior:
t+0: handler1 has started (for fieldB)
t+60: handler1 has finished (for fieldB)
t+61: handler2 has started (for fieldB)
t+62: handler2 has finished (for fieldB)
t+63: handler1 has started (for fieldA)
t+64: handler1 has finished (for fieldA)

Actual behavior:
t+0: handler1 has started (for fieldB)
t+60: handler1 has finished (for fieldB)
t+61: handler2 has started (for fieldB and fieldA <- this may be ok)
t+62: handler2 has finished (for fieldB and fieldA <- this may be ok)
handler1 for fieldA is not starting (unexpected)

Kopf version

1.36.2

Kubernetes version

1.27

Python version

3.11

Code

@kopf.on.update()
async def on_a_b_update()
    await asyncio.sleep(60)

@kopf.on.field(field="spec.fieldB")
async def on_b_update()
    logger.info("ok")

Logs

No response

Additional information

What is interesting, if we submit update for fieldA first and then for fieldB - then everything is working.
The order of the handlers in the code and the order of events submitted matters for reproducing.
The test case is reproducible in 100% of the time.

Potential Root Cause

The first event (for fieldB) should be consumed by 2 handlers. By the time the second handler fires (which has a filter for fieldB), the event for fieldA has arrived - now they are both consumed by second handler (which may be ok) but this is causing the first handler not to fire for fieldA (maybe because the first handler has the order before the second one)