Unable to pull helm-chart
boehb-efs opened this issue · comments
Describe the bug
Helm-chart is no longer available.
To Reproduce
> helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
> helm repo update
> helm pull application-gateway-kubernetes-ingress/ingress-azure
Error: failed to fetch https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/ingress-azure-1.7.5.tgz : 409 Public access is not permitted on this storage account.
https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/index.yaml
I'm having the same issue and this is now completely blocking production deployments.
Is there a mirror repo available?
This also is affecting all of our deployments and thus affecting ability to promote changes to production.
Is there anyway to raise the priority of this ticket?
Are we at the mercy of waiting for our American counterparts to come on line?
Is there a way to work out who or which team owns the storage account? This change might not (quite likely) be related to the owners of this repo if it was a security teams decision to lock down storage accounts.
I had the chart in my local cache which I've uploaded to a temp repo.
https://github.com/abunnyuk/ingress-azure-temp
Hopefully it's of use to someone in the short term.
Im also blocked because of this. Any updates?
This is working now!
This is working now!
Still facing the same issue. Are you using https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/?
Not fixed. Still an issue here
This is working now!
Still not working for me. Are you using the same repo link here?
Still not working for me.
This is working now!
I think this response is for
I had the chart in my local cache which I've uploaded to a temp repo.
https://github.com/abunnyuk/ingress-azure-temp
Hopefully it's of use to someone in the short term.
Any response here yet? This is a big issue for anyone relying on this chart for production!
Hi, we are running into this issue as well. Hope that it gets fixed ASAP since it's blocking our developments and preventing us to deploy to production. :-(
Same issue here, it's blocking us and we need it to be resolved.
Same as all the above having issues.
Blocked from the daily dev flow and as we currently are in a release phase, we cannot get anything out for validation in our staging environments.
But most importantly we cannot get anything out to our production environment.
This has impact for our business, dev- and test teams and end users - needs escalation.
And gone again for me also. The index file is now available for me. But the following is still blocked:
https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/ingress-azure-1.7.5.tgz
At 6pm UK time yesterday when I posed that it worked. It did, but looks like it was temp.
I raised a support call yesterday with Microsoft and I have a call with them about this issue in 30 minutes hopefully. Will see if the team I get can do anything.
Still not working for me.
This is working now!
I think this response is for
I had the chart in my local cache which I've uploaded to a temp repo.
https://github.com/abunnyuk/ingress-azure-temp
Hopefully it's of use to someone in the short term.
Unfortunately not. I hadn't gone down that route yet. This was a genuine "it did work as expected at the correct url" - but now blocked again. :-(
I quickly updated my pipeline to check if we still get a 409, then use the template kindly provided by @abunnyuk. Thanks
I think that will speed up our Ingress controller migration.
I quickly updated my pipeline to check if we still get a 409, then use the template kindly provided by @abunnyuk. Thanks I think that will speed up our Ingress controller migration.
I'm so glad it's proved useful to someone.
If it's of further use to anyone, I can also provide the Bicep and post-deployment commands for the AGIC Helm deployment with a shared Application Gateway, the purpose of which is to get mutual TLS working without having to move to Application Gateway for Containers.
I have spoken to them, but as I raised the request via the portal, the person could only really help with issues with using the cli or the portal. However, he did say that he would try to find the team looking after the repo/storage account and try their best to get some help.
I will also try out Microsoft internal partners once again.
Hello, @akshaysngupta @draychev @3quanfeng - is anyone on the team aware of this issue?
I also raised a ticket to MSFT but they only replied "they're working on it" so far.
Why is there no update on this from any official channel from MS?
Why has the underlying change even propagated in this manner?
"Working on it" on individual contact is not enough at this stage.
That's what I got just now:
After internal engagement we have the feedback from PM that they are actively working on this issue since we have our customers impacted.
I understand your needs and limitations though for now the workaround that we have from our end while they are working on this incident:
https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-install-existing#example-scenario
Please provide our internal teams more time regarding this issue of kindly follow the action plan shared since we don’t have an ETA to provide yet.
Haven't tried the workaround yet however
Hey folks -- we were aware of this issue yesterday and have remediated this issue. You should be able to successfully deploy again.
We'll have another update in the coming days to prevent this issue from happening again.
Appreciate your patience!
Jack
Hello folks,
To further add to closure on this, our long term fix is completed with all new helm deployments referencing Microsoft Container Registry (MCR) vs a storage account. Documentation changes to reference the MCR endpoint are available both on https://azure.github.io/application-gateway-kubernetes-ingress/ and https://learn.microsoft.com/azure/application-gateway/ingress-controller-install-existing.
Jack
I used Chart.yaml
...
dependencies:
- name: ingress-azure
version: 1.7.5
repository: https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
and I don't have any good way to use the oci based chart.
You turned off access to https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/ recently and I don't think this is good especially because we lose access to
helm search repo
and your documentation https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-update-ingress-controller is still referencing it.
I used Chart.yaml
... dependencies: - name: ingress-azure version: 1.7.5 repository: https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/and I don't have any good way to use the oci based chart.
You turned off access to https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/ recently and I don't think this is good especially because we lose access to
helm search repoand your documentation https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-update-ingress-controller is still referencing it.
We are in the process of migrating to the oci:// endpoint (note that you need Helm 3.8 or above if you want official Helm support for OCI registries).
We are moving from the old configuration:
repositories:
- name: application-gateway-kubernetes-ingress
url: https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
releases:
- name: application-gateway-ingress-controller
chart: application-gateway-kubernetes-ingress/ingress-azure
version: 1.7.4
...
to a new configuration (notice we no longer define the repository, and just directly reference the OCI endpoint in the chart):
releases:
- name: application-gateway-ingress-controller
chart: oci://mcr.microsoft.com/azure-application-gateway/charts/ingress-azure
version: 1.7.5
...
This approach has been working for us & hopefully it's useful to people landing on this issue after Microsoft made the storage account private again.
FWIW, I spoke with Microsoft Support and was informed that as of right now, customers should expect the old storage account to remain private, since OCI is their recommended path forward. OCI is also used by the Helm charts for Application Gateway for Containers for anyone using that.
I see
...
dependencies:
- name: ingress-azure
version: 1.7.5
repository: oci://mcr.microsoft.com/azure-application-gateway/charts
works fine thanks