Getting panic for `metrics cadvisor` command
Dentrax opened this issue · comments
Summary
Provide brief overview and context for the discovered bug.
Steps to Reproduce
curl -LO https://github.com/cyberark/kubeletctl/releases/download/v1.11/kubeletctl_darwin_amd64 && chmod a+x ./kubeletctl_darwin_amd64 && mv ./kubeletctl_darwin_amd64 /usr/local/bin/kubeletctl
kubeletctl -s 10.11.12.13 metrics cadvisor
- See error
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1763c4e]
goroutine 1 [running]:
kubeletctl/cmd.PrintPrettyHttpResponse(0x0, {0xc0000f17d0?, 0xc00021fd48?})
/home/cyber/kubeletctl/cmd/print.go:56 +0x2e
kubeletctl/cmd/metrics.glob..func1(0x1f0d520?, {0x18c600e?, 0x4?, 0x4?})
/home/cyber/kubeletctl/cmd/metrics/cadvisor.go:45 +0x4f
github.com/spf13/cobra.(*Command).execute(0x1f0d520, {0xc000324f00, 0x4, 0x4})
/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:842 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x1f0cd40)
/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:943 +0x37d
github.com/spf13/cobra.(*Command).Execute(...)
/home/cyber/kubeletctl/vendor/github.com/spf13/cobra/command.go:883
kubeletctl/cmd.Execute()
/home/cyber/kubeletctl/cmd/root.go:83 +0x25
main.main()
/home/cyber/kubeletctl/main.go:20 +0x17
Expected Results
It should work?
Actual Results
Throws panic.
Reproducible
- Always
- Sometimes
- Non-Reproducible
Version/Tag number
1.11
K8s: v1.23.7
Environment setup
macOS
Additional Information
Add any other context about the problem here.
It should work, but even if not, it shouldn't throw panic.
I didn't test this binary in macOs, but I can check tomorrow, maybe I will be able to do it.
To troubleshoot I have a number of questions.
If you run the below command:
curl -k https://10.11.12.13:10250/metrics/cadvisor
Do you receive any results?
Another question,
If you run:
kubeletctl -s 10.11.12.13 healthz
Does it work?
I think you can't auth to kubelet from the local:
curl -s -k https://10.11.12.13:10250/metrics/cadvisor
Unauthorized
You should pass the necessary certs: (on remote host)
curl -s -k https://localhost:10250/metrics/cadvisor --cert /etc/kubernetes/ssl/apiserver-kubelet-client.crt --key /etc/kubernetes/ssl/apiserver-kubelet-client.key
healthz
is working as expected:
kubeletctl -s 10.11.12.13 healthz
[*] Using KUBECONFIG environment variable
[*] You can ignore it by modifying the KUBECONFIG environment variable, file "~/.kube/config" or use the "-i" switch
ok
Thank you for providing that information.
The healthz
API worked for you so there is no problem with the compilation of the binary.
But there is some problem with the /metrics/cadvisor
API.
In your example, you wrote the command like that:
kubeletctl -s 10.11.12.13 metrics cadvisor
You didn't specify any certificates so the result should also be Unauthorized
when using kubeletctl.
We will test this API again and update it here.
Small update.
When I tested it I canceled the AuthN and AuthZ just for tests:
sed -i '/^authentication:$/,/^ webhook:$/ s/^\(\s*enabled:\s*\)false/\1true/' /var/lib/kubelet/config.yaml
sed -i '/^authorization:$/,/^ webhook:$/ s/^\(\s*mode:\s*\)Webhook/\1AlwaysAllow/' /var/lib/kubelet/config.yaml
To restore:
sed -i '/^authentication:$/,/^ webhook:$/ s/^\(\s*enabled:\s*\)true/\1false/' /var/lib/kubelet/config.yaml
sed -i '/^authorization:$/,/^ webhook:$/ s/^\(\s*mode:\s*\)AlwaysAllow/\1Webhook/' /var/lib/kubelet/config.yaml
In my case, on Ubuntu 22.04 LTS the command worked:
./build/kubeletctl_linux_amd64 -i metrics cadvisor
I received all the information.
The question is if the problem is because of using it on macOs or your metrics\cadvisor data cause it.
We will keep investigating.
I will check if I can run it on macOs and test it.
@Dentrax
I was able to reproduce it in MacOs Intel.
What CPU you have? Intel\M1\M2 ?
You can run the commands:
sysctl -n machdep.cpu.brand_string
uname -a
I want to know if it happens on M1 or M2.
My next plan is to debug it on macOs.
Hey, sorry missed your comment. @g3rzi
Apple M1 Max
Darwin REDACTED 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000 arm64
Can you try to build the code with darwin/arm64
(following @Rajchowdhury420 PR #38) and then test it?
It should be OK now. If after building darwin/arm64 it doesn't work, let us know.
Thanks @g3rzi, are you going to cut a new version soon? It seems almost a year passed since last tag.
Yes, published now version 1.12:
https://github.com/cyberark/kubeletctl/releases/tag/v1.12