Error enabling cloudwatch metrics when running on k8s 1.13
niksrc opened this issue · comments
There is no handler for https://cloud.weave.works/k8s/v1.13+/
Agent Logs:
time="2019-08-22T10:33:35Z" level=error msg="Error while deploying cloudwatch manifest: error: unable to read URL \"https://cloud.weave.works/k8s/v1.13+/cloudwatch.yaml?aws-region=us-east-1&aws-secret=cloudwatch&aws-resources=classic-elb%2Crds&aws-config=cloudwatch&aws-config-uid=*****&aws-secret-uid=******\", server reported 404 Not Found, status code=404\nFull output:\nerror: unable to read URL \"https://cloud.weave.works/k8s/v1.13+/cloudwatch.yaml?aws-region=us-east-1&aws-secret=cloudwatch&aws-resources=classic-elb%2Crds&aws-config=cloudwatch&aws-config-uid=******&aws-secret-uid=******\", server reported 404 Not Found, status code=404"
time="2019-08-22T10:50:48Z" level=info msg="Applying cloudwatch manifest from: https://cloud.weave.works/k8s/v1.13+/cloudwatch.yaml?aws-region=us-east-1&aws-secret=cloudwatch&aws-resources=classic-elb%2Crds&aws-config=cloudwatch&aws-config-uid=********&aws-secret-uid=****"
time="2019-08-22T10:50:48Z" level=error msg="Error while deploying cloudwatch manifest: error: unable to read URL
Why did you think that url would work?
Remove the + - this should work:
https://cloud.weave.works/k8s/v1.13/cloudwatch.yaml?aws-region=us-east-1&aws-secret=cloudwatch&aws-resources=classic-elb%2Crds&aws-config=cloudwatch&aws-config-uid=*****&aws-secret-uid=******
@bboreham I don't know, that's the URL I found in the weave-agent logs, so there must be an issue in the agent.
Can you email support@weave.works with details of the instance and the full log, please.
@bboreham I have sent an email with the instance details and the agent log.
OK, I figured out the + sign is coming from the EKS Kubernetes control-plane, like this:
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-eks-5ac0f1", GitCommit:"5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1", GitTreeState:"clean", BuildDate:"2019-08-20T22:39:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-eks-5047ed", GitCommit:"5047edce664593832e9b889e447ac75ab104f527", GitTreeState:"clean", BuildDate:"2019-08-21T22:32:40Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
We'll adjust the back-end to cope.
The + is documented in Kubernetes, albeit buried quite well, also a note that the major and minor fields will be deprecated.
So this program should be amended to respect that.
@niksrc the back-end is no longer returning 404 on that URL - can you confirm your installation is working better?
@bboreham yes, the metrics are getting scrapped properly. Closing the issue now.
Reopening because we still need to change the launcher to understand + in version numbers.