AKS - `jspolicy` pod retries validating webhook update
infa-ddeore opened this issue · comments
after applying #26 (comment) on AKS cluster jspolicy pod is full of below logs, repeating continuously, functionally everything is fine, what could be the issue?
I1213 10:27:36.595144 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.617848 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.637093 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.660114 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.694569 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.746162 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
I1213 10:27:36.795385 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-jhwl8
these are ~20 logs per second so looks like those many calls are made by jspolicy pod per second
EKS and GKE doesn't have any issue
log when deleting the jspolicy object
I1213 11:05:41.866732 1 loghelper.go:34] jspolicy-controller: Update validating webhook pod-policy.example.com-dkwjx
E1213 11:05:41.886877 1 controller.go:302] controller-runtime: manager: reconciler group policy.jspolicy.com reconciler kind JsPolicy: controller: jspolicy: name pod-policy.example.com namespace : Reconciler error Operation cannot be fulfilled on jspolicies.policy.jspolicy.com "pod-policy.example.com": StorageError: invalid object, Code: 4, Key: /registry/policy.jspolicy.com/jspolicies/pod-policy.example.com, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 818d53ac-68d4-40a4-b9c6-3496f14d2e68, UID in object meta:
issue doesnt seem to be resolved with v0.2.0-beta.1
re-installed jspolicy on AKS cluster
$ helm -n jspolicy delete jspolicy
$ k apply -f https://raw.githubusercontent.com/loft-sh/jspolicy/5211a03e9258d2f9917da3f4511af3af77fe441a/chart/crds/crds.yaml
helm install jspolicy jspolicy -n jspolicy --create-namespace --repo https://charts.loft.sh --version=v0.2.0-beta.1
jspolicy went into loop after applying mutating webhook:
k apply -f https://raw.githubusercontent.com/loft-sh/jspolicy/main/examples/by-use-case/add-node-selector.yaml
logs:
I1219 06:15:40.048297 1 logr.go:249] jspolicy-controller: add-node-selector.example.com: reconcile started
I1219 06:15:40.088854 1 logr.go:249] jspolicy-controller: add-node-selector.example.com: reconcile started
I1219 06:15:40.089482 1 logr.go:249] jspolicy-controller: Patching mutating webhook add-node-selector.example.com-qlhgw with {"webhooks":[{"admissionReviewVersions":["v1"],"clientConfig":{"caBundle":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZQakNDQXlhZ0F3SUJBZ0lHRnNVeXdxRU1NQTBHQ1NxR1NJYjNEUUVCQ3dVQU1CTXhFVEFQQmdOVkJBb1QKQ0dwemNHOXNhV041TUI0WERUSXhNVEl4TmpFd01UQXdNbG9YRFRNeE1USXhOakV3TVRBd01sb3dFekVSTUE4RwpBMVVFQ2hNSWFuTndiMnhwWTNrd2dnSWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUNEd0F3Z2dJS0FvSUNBUUM2ClArRWV5dXdKL0xQRkQ5Z0NHOXlIV3hUMk1HOHk2ZXpnNlBBU2NGTXgwRFV2S1pzTDYwazdRYXhJdUtlMU1FY28KQXRJVjBKcndPbk5yNGMvNzZlOWpMQlpvTGNMY09PcjZ2YzNHQnpBR25DVkRKU2Q4YnM5RVFrWkp3ZW96alNZMwpsS1d4OVd0dGFjR0Zuci9KM1R2bndjdlkraEh5UklUWUxhMkdlZElwY01LOWFZQ1ZicnVKdjRNZFIyU3JVMjFDCisxaGFVSGxDMFRpL20xeW9VdFozNTZ4THZlZTVIQmJBMCtUdm95OVlmZjZCSGNxY0xiVlNZa3hUZloxSWw3a3MKUXpPZ0h1NXR4WkNycUdCRlQvdGFoMW5ZVmxTQ3RkQ2dSMlhCMzdabzRQclJaaUJVVVNyaGdoSlFJbU16c0RSaQpUcVd5eWZZOE9xZ1hrbkFSa3VHS1RQWUNTVjZTbFp1c1p2SDg0cFJMWVZ6cFl2Y3VQbXVFMEhncWNrSGhnZ2l6CkYzNXVFbXdCNHZHeVNFVCtkVkZvSXFBa3I4cWV4SzBqeUFEY2xuNTNmbGNyc05xVVlxVmd2SVA2VDF5eDFOb3MKaXpIUUZOMnpoTkl4RDNFY3ZERzhzOXg5SGZIdVN4aUlYa2FKTGZrc2lGRWsxRjhWRDN6clRhNzBSakoyeVlzdwprK0VHNmM2bENReDMzSy8va1RzVzl2KzVLU0pCRXZlb0VxT0tFbDdSb1VrVzcrY3pxUE5xRmVCdmIrTi9MZ2VwCmw2L1BMZjVIeS9ycE9HWlZoTmh4Y3pZdW1LdkpIeDFjOG9XbFc5QjZGQkpGeW5vZzU4SFJHd0hXQk0xVmZDdEQKOFNxWXMyN1FVbFEwNU5oMyttRTJVbkpWeEhRREJBaFRLSnF0NmtNWVJRSURBUUFCbzRHWE1JR1VNQTRHQTFVZApEd0VCL3dRRUF3SUhnREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQWdZSUt3WUJCUVVIQXdFd0RBWURWUjBUCkFRSC9CQUl3QURBT0JnTlZIUTRFQndRRkFRSURCQVl3UlFZRFZSMFJCRDR3UElJVmFuTndiMnhwWTNrdWFuTncKYjJ4cFkza3VjM1pqZ2lOcWMzQnZiR2xqZVM1cWMzQnZiR2xqZVM1emRtTXVZMngxYzNSbGNpNXNiMk5oYkRBTgpCZ2txaGtpRzl3MEJBUXNGQUFPQ0FnRUFTUUFCQzZIY1dmaEg2NS82L0dXQm1QQWVraUZ4OHd1VFVsQUM5TVJ6CmtBQ3NJNThOTG9nVjF5WTdna2NyN0tXMnoyNDJxcmk3b0pnQzJGZmc1d3FHRFpNRnN3WHVDNG9Odkd6K3R2elUKY0M5VHZKQVhYZ0w5eW1FQ1dlblFyazZIaU1BQ1JabGtRRXBaTFFDeFFRNmJoQk1OTERSZTFqQTdZazJSRmFELwpHaVV5T3pVT2hheFdRbnZTb1N0Y2pXMWR4WTh5VW1aZ3VLT21raFFJV0ZBRnpQbWNFQ2E4VExRMFY2TXI3amZZCk14L0lsd2pncGV1U0J3bCtBeW1iOTJjTnoyVGhZdW16TmNDOTFUTTVIREk3L1JXNUJNQjF1SjYxUkVZVjhUdTgKc0hLamRhTUtPa3VOTTVNdyt0UlpMOFpsTkEwdUpSa04vQVZiN2NqQ0xsTTMzaGgvcHZnRzN3MzQ0WFF4Qzg0cwprNUozTkxpVXR1UXIwZU03VjlpRmNUL0dvWU1qNTk4cXRpc0NUOWJnUDBaWitBTXplb0oyT3ZpMWdOd1JFSi9KCko2a3FWaXI5TVJxQTZuVU05Z21nZmZjRHBEVmYva3RvTkpuV3hzQ1dPbmNZdm5tTGVHKzhOUFgwTFpSRjRtUisKM2piVm9xeENPZmlkVkJPam5PaldUQ1lRRWFaZ1dCcUY2SkhxbDV1T29RSmN1RVd2Um9YWnU0Z2Z2cGF6YVhCVwpXSXlhZE04ajhFSWRrbEtiejFiQlpJV1JTb05WSDhGQWtHOGdvbnFUdXlrV3lHS25JR1JWY05SbjhIVHp0RGhoCk5UVzQ3dzNvaHJ6aDJFV05jVS9OaWVYeDU3VkhnaU5aOFNGaDZnVjVZbit1azdIczFWSHlXei8rSWNHYS9oMHcKWENnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","service":{"name":"jspolicy","namespace":"jspolicy","path":"/policy/add-node-selector.example.com","port":443}},"failurePolicy":"Fail","matchPolicy":"Equivalent","name":"add-node-selector.example.com","namespaceSelector":{"matchLabels":{"with-node-selector":"true"}},"objectSelector":{},"reinvocationPolicy":"Never","rules":[{"apiGroups":["*"],"apiVersions":["*"],"operations":["CREATE"],"resources":["pods"],"scope":"*"}],"sideEffects":"None","timeoutSeconds":10}]}
I1219 06:15:40.108060 1 logr.go:249] jspolicy-controller: add-node-selector.example.com: reconcile started
I1219 06:15:40.137745 1 logr.go:249] jspolicy-controller: add-node-selector.example.com: reconcile started
I1219 06:15:40.138196 1 logr.go:249] jspolicy-controller: Patching mutating webhook add-node-selector.example.com-qlhgw with {"webhooks":[{"admissionReviewVersions":["v1"],"clientConfig":{"caBundle":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZQakNDQXlhZ0F3SUJBZ0lHRnNVeXdxRU1NQTBHQ1NxR1NJYjNEUUVCQ3dVQU1CTXhFVEFQQmdOVkJBb1QKQ0dwemNHOXNhV041TUI0WERUSXhNVEl4TmpFd01UQXdNbG9YRFRNeE1USXhOakV3TVRBd01sb3dFekVSTUE4RwpBMVVFQ2hNSWFuTndiMnhwWTNrd2dnSWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUNEd0F3Z2dJS0FvSUNBUUM2ClArRWV5dXdKL0xQRkQ5Z0NHOXlIV3hUMk1HOHk2ZXpnNlBBU2NGTXgwRFV2S1pzTDYwazdRYXhJdUtlMU1FY28KQXRJVjBKcndPbk5yNGMvNzZlOWpMQlpvTGNMY09PcjZ2YzNHQnpBR25DVkRKU2Q4YnM5RVFrWkp3ZW96alNZMwpsS1d4OVd0dGFjR0Zuci9KM1R2bndjdlkraEh5UklUWUxhMkdlZElwY01LOWFZQ1ZicnVKdjRNZFIyU3JVMjFDCisxaGFVSGxDMFRpL20xeW9VdFozNTZ4THZlZTVIQmJBMCtUdm95OVlmZjZCSGNxY0xiVlNZa3hUZloxSWw3a3MKUXpPZ0h1NXR4WkNycUdCRlQvdGFoMW5ZVmxTQ3RkQ2dSMlhCMzdabzRQclJaaUJVVVNyaGdoSlFJbU16c0RSaQpUcVd5eWZZOE9xZ1hrbkFSa3VHS1RQWUNTVjZTbFp1c1p2SDg0cFJMWVZ6cFl2Y3VQbXVFMEhncWNrSGhnZ2l6CkYzNXVFbXdCNHZHeVNFVCtkVkZvSXFBa3I4cWV4SzBqeUFEY2xuNTNmbGNyc05xVVlxVmd2SVA2VDF5eDFOb3MKaXpIUUZOMnpoTkl4RDNFY3ZERzhzOXg5SGZIdVN4aUlYa2FKTGZrc2lGRWsxRjhWRDN6clRhNzBSakoyeVlzdwprK0VHNmM2bENReDMzSy8va1RzVzl2KzVLU0pCRXZlb0VxT0tFbDdSb1VrVzcrY3pxUE5xRmVCdmIrTi9MZ2VwCmw2L1BMZjVIeS9ycE9HWlZoTmh4Y3pZdW1LdkpIeDFjOG9XbFc5QjZGQkpGeW5vZzU4SFJHd0hXQk0xVmZDdEQKOFNxWXMyN1FVbFEwNU5oMyttRTJVbkpWeEhRREJBaFRLSnF0NmtNWVJRSURBUUFCbzRHWE1JR1VNQTRHQTFVZApEd0VCL3dRRUF3SUhnREFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQWdZSUt3WUJCUVVIQXdFd0RBWURWUjBUCkFRSC9CQUl3QURBT0JnTlZIUTRFQndRRkFRSURCQVl3UlFZRFZSMFJCRDR3UElJVmFuTndiMnhwWTNrdWFuTncKYjJ4cFkza3VjM1pqZ2lOcWMzQnZiR2xqZVM1cWMzQnZiR2xqZVM1emRtTXVZMngxYzNSbGNpNXNiMk5oYkRBTgpCZ2txaGtpRzl3MEJBUXNGQUFPQ0FnRUFTUUFCQzZIY1dmaEg2NS82L0dXQm1QQWVraUZ4OHd1VFVsQUM5TVJ6CmtBQ3NJNThOTG9nVjF5WTdna2NyN0tXMnoyNDJxcmk3b0pnQzJGZmc1d3FHRFpNRnN3WHVDNG9Odkd6K3R2elUKY0M5VHZKQVhYZ0w5eW1FQ1dlblFyazZIaU1BQ1JabGtRRXBaTFFDeFFRNmJoQk1OTERSZTFqQTdZazJSRmFELwpHaVV5T3pVT2hheFdRbnZTb1N0Y2pXMWR4WTh5VW1aZ3VLT21raFFJV0ZBRnpQbWNFQ2E4VExRMFY2TXI3amZZCk14L0lsd2pncGV1U0J3bCtBeW1iOTJjTnoyVGhZdW16TmNDOTFUTTVIREk3L1JXNUJNQjF1SjYxUkVZVjhUdTgKc0hLamRhTUtPa3VOTTVNdyt0UlpMOFpsTkEwdUpSa04vQVZiN2NqQ0xsTTMzaGgvcHZnRzN3MzQ0WFF4Qzg0cwprNUozTkxpVXR1UXIwZU03VjlpRmNUL0dvWU1qNTk4cXRpc0NUOWJnUDBaWitBTXplb0oyT3ZpMWdOd1JFSi9KCko2a3FWaXI5TVJxQTZuVU05Z21nZmZjRHBEVmYva3RvTkpuV3hzQ1dPbmNZdm5tTGVHKzhOUFgwTFpSRjRtUisKM2piVm9xeENPZmlkVkJPam5PaldUQ1lRRWFaZ1dCcUY2SkhxbDV1T29RSmN1RVd2Um9YWnU0Z2Z2cGF6YVhCVwpXSXlhZE04ajhFSWRrbEtiejFiQlpJV1JTb05WSDhGQWtHOGdvbnFUdXlrV3lHS25JR1JWY05SbjhIVHp0RGhoCk5UVzQ3dzNvaHJ6aDJFV05jVS9OaWVYeDU3VkhnaU5aOFNGaDZnVjVZbit1azdIczFWSHlXei8rSWNHYS9oMHcKWENnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","service":{"name":"jspolicy","namespace":"jspolicy","path":"/policy/add-node-selector.example.com","port":443}},"failurePolicy":"Fail","matchPolicy":"Equivalent","name":"add-node-selector.example.com","namespaceSelector":{"matchLabels":{"with-node-selector":"true"}},"objectSelector":{},"reinvocationPolicy":"Never","rules":[{"apiGroups":["*"],"apiVersions":["*"],"operations":["CREATE"],"resources":["pods"],"scope":"*"}],"sideEffects":"None","timeoutSeconds":10}]}
changed jspolicy deployment replicas to zero to let AKS apply the changes to mutating webhook, attaching jspolicy.yaml.txt and aks.yaml.txt files
jspolicy.yaml.txt is the file updated by jspolicy pod, aks.yaml.txt is after jspolicy pod is scaled down to zero
@infa-ddeore thanks for the update, would be great if you could test this again with the new beta version v0.2.0-beta.3
verified on EKS/ AKS and GKE clusters
helm install jspolicy jspolicy -n jspolicy --create-namespace --repo https://charts.loft.sh --set image=my-repo.example.com/jspolicy:0.2.0-beta.4 --set imagePullSecrets[0].name=my-docker-secret --version=v0.2.0-beta.4
there are no loops now in any of these cloud's clusters, tested with validating and mutating jspolicies