Enable operator to scope based on namespace labels

[Jacob Delorme]

Problem

Enable watching namespaces based on their labels similar to how web hooks can be configured to serve namespaces based on their labels. Ideally this operator run would be treated as a cluster level operator so as to prevent the requirement of updating and requiring locks on N peering objects during operation.

Proposal

A few things i have tried already that do not quite work:

Granting the operator permissions to only the target namespaces with a ClusterRole and Rolebindings per namespace

For this to work you need to supply a pattern to the operator. Depending on namespace name patterns a pattern that matches only what you want may be impossible, selecting namespaces that are not a target. This causes the operator to run in namespace mode and require a peering on every namespace it can see. The operator will be unable to a peering in namespaces it does not have permissions for which effectively stops the operator from ever running.

Update the operator with specific namespaces

This is currently the only plausible workaround. This is not ideal as it requires a roll of the operators on every namespace change which can happen relatively frequently.

Ideas

This could be implemented by allowing a labels flag at startup (--namespace-labels?) and applying it on the query used to list/watch namespaces. This would also need to be applied to the Webhook configs if those are created automatically as well.

Another potential option is to accept the same flag but let kopf do the filtering in revise_namespaces.

I am open to writing up a PR but wanted to get some feedback on what would be the best approach or of any gotchas i am not seeing.

[Steven Armstrong]

Not sure I fully understand what you are after.
But could something like this work?
Untested code, but you get the idea.

@kopf.index('', 'v1', 'namespace', labels={'some-label': kopf.PRESENT})
def labeled_namespaces(name, **_): 
    return name
    
def filter_by_namespaces(namespace, labeled_namespaces: kopf.Index, **_):
    # Return True if we care about this namespace, False otherwise.
    return namespace in labeled_namespaces
    
@kopf.on.create('kopfexamples', labeled_namespaces: kopf.Index, when=filter_by_namespaces):
def my_handler(**_):
    # Do something with the kopfexample resource which is in a labeled namespace
    pass

[Sergey Vasilyev]

@asteven That might work as long as the operator has a read/watch permission to the whole cluster.

I guess, the problem is that it is not the case. The operator probably has permissions only to specific namespaces. As such, it should watch the stream on those specific namespaces only as if it were a namespaced operator. This can be partially done by namespace globs (inclusion & exclusion) like kopf run … -n *myop-*,!*exclude* -n *2ndop*. But that does not allow label-based filtering, or any runtime condition evaluation.

The machinery of starting & stopping the stream-watchers is hidden inside the framework (observation.py + orchestration.py) and does not let any user-side control of it.

I currently think on what could be a good API to expose such control — especially taking the requests to dynamically control the resource kinds too (as in: one known resource kind X lists other resource kinds A, B, C, which actually should be operated).

[Jacob Delorme]

I was able to implement @asteven suggestion by also mixing the index filter with the per namespace permissions. As @nolar pointed out i had to grant global read to all managed resources at the cluster level which is not ideal.