Embedding in shared event loop
scabala opened this issue · comments
Keywords
embed
Problem
First of all - thank you for this project. Having framework for building operators in native Python is dope!
I have a question about embedding kopf.
Documentation states that kopf needs its own event loop because it considers all spawned tasks as its own. That would interfere with different application. This requirement seems like something coming from kopf itself rather then from asyncio or Python.
The question: why?
For me, it seemed obvious to run some web app with embedded operator on the same event loop. Requirement to have separate event loop seems strange.
Maybe I'm naive but maybe it could be solved by:
a) keeping reference to all kopf-spawned tasks and cancel only them when embedding
b) set some custom attribute (i.e. __kopf_managed__) on created tasks and when shutting down embedded operator, just filter for tasks with such attribute
I do not mean to be mean or critical - I am genuinely curious what is the reason for this requirement.
Hi. You are welcome!
keeping reference to all kopf-spawned tasks
Exactly this is a problem — it is difficult to achieve. First, Kopf spawns tasks for its own needs somewhere deep in the call stack, some of them are "fire-and-forget" style (daemons, timers). Second, regular handlers can spawn new tasks — this was the case before daemons/timers arrived, so users had to spawn their own tasks explicitly in the resume/creation handlers.
All in all, at the end, when the operator exits, it must ensure the graceful termination of all spawned tasks/daemons/timers; here, "ensure" means 2 steps: cancellation + awaiting (actually, 2 cancellation attempts: soft & hard, each with waiting), plus also some logging on the problems if some tasks do not end and are "abandoned".
Since it is difficult, the task tracking is instead delegated to the event loop, which already has this functionality.
So, Kopf considers all tasks spawned after its startup as its own — for simplicity.
For the advanced cases if someone wants to run it all in a shared event loop or orchestrate the tasks themselves, there is kopf.spawn_tasks() as the official Kopf API. You can replace the kopf.operator() call with the call to kopf.spawn_tasks() and then orchestrate them somehow the same way as in kopf.run_tasks(). The logic is simple: wait for the spawned tasks infinitely; once any single one of them exits, terminate others as gracefully as possible — nothing more. What "other tasks" mean in a shared loop — it is for you to decide.
set some custom attribute…
The official asyncio API does not provide this capability, so you cannot be sure that setting arbitrary attributes on tasks is possible. Tasks can be slotted or C-level objects with no __dict__. Or they can be tasks from other event loop implementations — e.g. uvloop — which also cannot accept arbitrary attributes. So, such tracking must be done in a centralized place, such as an operator-scoped weakref.WeakSet or alike.
it seemed obvious to run some web app with embedded operator on the same event loop
That is a somewhat debatable and rather controversial point of view. I personally would keep separate components of an application in separate threads, each having its event loop. Cross-thread communication is rather safe & fast and does not add any extra overhead (as inter-process communication does, for example). Since both apps are i/o-bound, GIL is also not a problem. But I prefer not to argue about how other people do this in their apps — because "whatever works, works".
At the end of the day, such task-tracking capabilities can be considered and added, but that would require a huge refactoring and a lot of effort with little or no benefit for the majority of use cases (compared to possible workarounds: multi-threading or using kopf.spawn_tasks()).
Sadly, I am not able to dedicate enough time to this project nowadays, so it remains in its current state (only some major bugs are fixed).
Thanks for reply!
So, Kopf considers all tasks spawned after its startup as its own — for simplicity.
I see, that is understandable.
I have come up with another idea - instead of keep references to all tasks in one place (as you mentioned that would require heavy refactoring), name them in specific way, i.e. with prefix kopf.. That would obviously work only for kopf tasks and might be extended to tasks created via kopf.spawn_tasks or similar. If user of kopf spawns task using different mechanism - no way to manage it. I know, sounds brittle but I wanted to share the idea nonetheless.
For the advanced cases if someone wants to run it all in a shared event loop or orchestrate the tasks themselves, there is kopf.spawn_tasks() as the official Kopf API. You can replace the kopf.operator() call with the call to kopf.spawn_tasks() and then orchestrate them somehow the same way as in kopf.run_tasks(). The logic is simple: wait for the spawned tasks infinitely; once any single one of them exits, terminate others as gracefully as possible — nothing more. What "other tasks" mean in a shared loop — it is for you to decide.
That sounds very interesting, I'll need to look into this. Maybe it could be documented under section I want to embed kopf but still use shared event loop?
That is a somewhat debatable and rather controversial point of view. I personally would keep separate components of an application in separate threads, each having its event loop. Cross-thread communication is rather safe & fast and does not add any extra overhead (as inter-process communication does, for example). Since both apps are i/o-bound, GIL is also not a problem. But I prefer not to argue about how other people do this in their apps — because "whatever works, works".
That's a matter of preference. For me one thing was obvious, for someone else - something entirely different. Different use-cases means different requirements, etc. Still, would be nice have ability to run kopf in shared event loop. Will try that and see how it works - maybe even craft a PR with documentation, who knows.
Thanks for pointers!
Task naming is available since Python 3.8. Kopf still supports Python 3.7. I’m eagerly waiting for the end-of-life for 3.7 as a milestone, which happens somewhen this summer 2023 if I remember correctly.
For anyone seeing this thread, there are some docs for this now: https://kopf.readthedocs.io/en/stable/embedding/