Upgrade fails due to existing canary ingress
kamalraj-asurion opened this issue · comments
Describe the bug
We have flux and helm-operator on our cluster along with flagger for canary deployments.
flux:1.19.0
helm-operator:1.2.0
flagger:1.6.0
EKS Cluster: 1.18
After the recent upgrade to EKS 1.18 and nginx ingress (ingress-nginx/controller:v0.45.0), we are seeing issues with upgrades where they are failing with the message
Upgrade "test-deploy" failed: cannot patch "test-deploy" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "test-deploy.aws.net" and path "/" is already defined in ingress test-ns/test-deploy-canary
This causes the HelmRelease to be in Failed state and hence not allowing any further upgrade.
right now I have disabled the admissionWebhooks on the ingress-nginx deployment to proceed on the upgrades. not sure if it is the right way to proceed on it.
Can you please guide on a better way to handle this or correct me if i am making any mistake on the dependencies (version mismatch between the components) or something else is the cause of this issue.
To Reproduce
Steps to reproduce the behaviour:
With EKS 1.18 and ingress-nginx/controller:v0.45.0 i was able to see this issue with below steps
install a new component (HelmRelease with flagger enabled)
update the image and let it upgrade (Fails here due to ingress)
Expected behavior
Since there are 2 ingress in this case (canary - flag being set to false as the ramp up is complete from previous install/upgrade), i would expect things to be handled gracefully and proceed with the upgrade.
Logs
ts=2021-06-01T16:36:55.248390515Z caller=helm.go:69 component=helm version=v3 info="error updating the resource "test-deploy":\n\t cannot patch "test-deploy" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "test-deploy.net" and path "/" is already defined in ingress test-ns/test-deploy-canary" targetNamespace=test-ns release=test-deploy
Additional context
- Helm Operator version: 1.2.0
- Kubernetes version: EKS (1.18)
- Git provider:
- Helm repository provider:
Sorry if your issue remains unresolved. The Helm Operator is in maintenance mode, we recommend everybody upgrades to Flux v2 and Helm Controller.
A new release of Helm Operator is out this week, 1.4.4.
We will continue to support Helm Operator in maintenance mode for an indefinite period of time, and eventually archive this repository.
Please be aware that Flux v2 has a vibrant and active developer community who are actively working through minor releases and delivering new features on the way to General Availability for Flux v2.
In the mean time, this repo will still be monitored, but support is basically limited to migration issues only. I will have to close many issues today without reading them all in detail because of time constraints. If your issue is very important, you are welcome to reopen it, but due to staleness of all issues at this point a new report is more likely to be in order. Please open another issue if you have unresolved problems that prevent your migration in the appropriate Flux v2 repo.
Helm Operator releases will continue as possible for a limited time, as a courtesy for those who still cannot migrate yet, but these are strongly not recommended for ongoing production use as our strict adherence to semver backward compatibility guarantees limit many dependencies and we can only upgrade them so far without breaking compatibility. So there are likely known CVEs that cannot be resolved.
We recommend upgrading to Flux v2 which is actively maintained ASAP.
I am going to go ahead and close every issue at once today,
Thanks for participating in Helm Operator and Flux! 💚 💙