Use nodeSelector for otomi workloads instead of Gatekeeper mutation webhook

[jeho]

WHY

Currently the mutations.gatekeeper.sh/v1alpha1 is used to manipulate node placement.
It modifies the podAffinity, which is tricky. We also cannot target simply all pods that do not belong to the team namespace because

1. customers can have their own workloads out of team ns
2. Pods that spawned by DaemonSets (e.g.: promtail) should not be included by the mutation.

Due to that limitation, pods scheduled by operators often do not end up in the right node.

There is a simple and explicit way to instruct Pods to be scheduled in right place: nodeSelector and it is supported by all charts. For various CRs use nodePlacement to instruct controller to created pods with the nodeSelector spec set.

User Story

AS Aplatform admin
I WANT TOall Otomi pods to be scheduled in the right nodes
SOI have HW isolation between platform and application workloads

Acceptance criteria

GIVENnodeSelector property
WHENI upgrade Otomi to a new version
THENPods are using nodeSelector instead of podAffinity
AND Mutating webhook only applies to team namespaces.

Definition of done

• Relevant PRs are merged
• Tested by peer
• Updated documentation reviewed by peer
• Short demo video recorded and stored on google drive (if applicable)