kubectl exec and kubectl logs don't work in the tenant master in the demo
yaoyao12138 opened this issue · comments
Descriptions
kubectl exec and kubectl logs don't work in the tenant master in the Github demo
Error Output
➜ virtualcluster kubectl exec -it "${VC_POD}" --kubeconfig vc-1.kubeconfig -- /bin/sh
Error from server:
➜ virtualcluster kubectl logs "${VC_POD}" --kubeconfig vc-1.kubeconfig
Error from server (NotFound): the server could not find the requested resource ( pods/log test-deploy-5f4bcd8c-fplld)
Logs
Add - --v=4 in the container command/args list of the vnagent daemonset template directly.
Here is one of the vn-agent pods value log content.
➜ virtualcluster oc logs vn-agent-xx6lj -n vc-manager
I0407 07:48:15.423006 1 cert.go:61] Using self-signed cert (/etc/vn-agent/vn.crt, /etc/vn-agent/vn.key)
I0407 07:48:15.423058 1 server.go:123] server listen on :10550
I0407 07:49:49.682306 1 route.go:124] request /exec/default/test-deploy-5f4bcd8c-fplld/poc?command=sh&input=1&output=1&tty=1
I0407 07:49:49.682453 1 route.go:128] will forward request to kubelet
I0407 07:49:49.682472 1 route.go:141] request after translate https://127.0.0.1:10250/exec/default-e66be9-vc-sample-1-default/test-deploy-5f4bcd8c-fplld/poc?command=sh&input=1&output=1&tty=1
Other Output
The kubectl exec and kubectl logs worked in the super master, and also the tenant master worked.
➜ virtualcluster kubectl exec -it "${VC_POD}" -n default-e66be9-vc-sample-1-default sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ # ls
bin dev etc home proc root sys tmp usr var
➜ virtualcluster kubectl logs "${VC_POD}" -n default-e66be9-vc-sample-1-default
Mem: 16043088K used, 213652K free, 1809624K shrd, 3144K buff, 11487344K cached
CPU: 10.9% usr 9.3% sys 0.0% nic 76.4% idle 0.4% io 1.0% irq 0.4% sirq
Load average: 2.53 3.23 3.12 5/1850 21
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
➜ virtualcluster kubectl get po ${VC_POD} --kubeconfig vc-1.kubeconfig
NAME READY STATUS RESTARTS AGE
test-deploy-5f4bcd8c-fplld 1/1 Running 0 18h
Env Info
➜ virtualcluster kind version
kind v0.10.0 go1.15.7 linux/amd64
➜ virtualcluster kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2-0-g52c56ce", GitCommit:"6082e941e6d62f3a0c6ca8ba52927100948b1d0d", GitTreeState:"clean", BuildDate:"2020-11-14T01:08:04Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-21T01:11:42Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
➜ virtualcluster cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)
In a minikube cluster, the kubectl exec and kubectl logs don't work in the tenant master either.
➜ virtualcluster kubectl exec -it test-deploy-5f4bcd8c-nfvq4 sh --kubeconfig vc-1.kubeconfig
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
error: unable to upgrade connection: Unauthorized
➜ virtualcluster kubectl logs test-deploy-5f4bcd8c-nfvq4 --kubeconfig vc-1.kubeconfig
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log test-deploy-5f4bcd8c-nfvq4))
➜ virtualcluster minikube version
minikube version: v1.18.1
commit: 09ee84d530de4a92f00f1c5dbc34cead092b95bc
Are you trying the optional following requests to Kubelet approach? Can you try the default mode which forwards the requests to super cluster apiserver and see if that works?
@Fei-Guo I tried the requests to super cluster apiserver in the Kind cluster and Minikube cluster, and they both work.
➜ virtualcluster kubectl exec -it "${VC_POD}" -n default-e66be9-vc-sample-1-default sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ # ls
bin dev etc home proc root sys tmp usr var
➜ virtualcluster kubectl logs "${VC_POD}" -n default-e66be9-vc-sample-1-default
Mem: 16043088K used, 213652K free, 1809624K shrd, 3144K buff, 11487344K cached
CPU: 10.9% usr 9.3% sys 0.0% nic 76.4% idle 0.4% io 1.0% irq 0.4% sirq
Load average: 2.53 3.23 3.12 5/1850 21
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
Sorry，I did not explain it clearly.
If you don't provide KubeletClientCert in the vn-agent start up command by removing
" - --cert-dir=/etc/vn-agent/
- --kubelet-client-certificate=/etc/vn-agent/pki/client.crt
- --kubelet-client-key=/etc/vn-agent/pki/client.key"
from container cmd args, the kubelet requests will be forwarded to supercluster apiserver.
You should be able to see "will forward request to super apiserver" in the vn-agent log instead of "will forward request to kubelet".
If this method works, it is possible that you are not preparing the kubelet cert correctly following the demo doc. It is also likely the demo doc has some bugs.
@Fei-Guo It seems actually the problems due to the kubelet cert. Due to I try the way by removing
" - --cert-dir=/etc/vn-agent/
- --kubelet-client-certificate=/etc/vn-agent/pki/client.crt
- --kubelet-client-key=/etc/vn-agent/pki/client.key"
from container cmd args, I could see "will forward request to super apiserver" in the vn-agent log instead of "will forward request to kubelet".
And I check the ~/.minikube/, tried the 3 pairs of key/cert, all don't not work which make me confused ..
➜ .minikube ls -al ~/.minikube
total 40
drwxr-xr-x 10 root root 307 Apr 12 01:44 .
dr-xr-x---. 24 root root 4096 Apr 12 02:03 ..
drwxr-xr-x 2 root root 6 Mar 31 00:56 addons
-rw-r--r-- 1 root root 1111 Mar 31 01:18 ca.crt
-rw------- 1 root root 1675 Mar 31 01:18 ca.key
-rwxr-xr-x 1 root root 1074 Apr 8 22:12 ca.pem
drwxr-xr-x 4 root root 42 Mar 31 01:16 cache
-rwxr-xr-x 1 root root 1115 Apr 8 22:12 cert.pem
drwxr-xr-x 2 root root 69 Mar 31 01:17 certs
-rwxr-xr-x 1 root root 1115 Apr 12 01:58 client.crt
-rwxr-xr-x 1 root root 1679 Apr 12 01:58 client.key
drwxr-xr-x 2 root root 25 Apr 1 23:52 config
drwxr-xr-x 2 root root 6 Mar 31 00:56 files
-rwxr-xr-x 1 root root 1679 Apr 8 22:12 key.pem
drwxr-xr-x 2 root root 45 Mar 31 01:18 logs
-rw------- 1 root root 0 Mar 31 01:17 machine_client.lock
drwxr-xr-x 7 root root 127 Apr 8 22:12 machines
drwxr-xr-x 8 root root 98 Apr 8 22:12 profiles
-rw-r--r-- 1 root root 1119 Mar 31 01:18 proxy-client-ca.crt
-rw------- 1 root root 1675 Mar 31 01:18 proxy-client-ca.key
Error 3 outputs:
➜ .minikube kubectl --kubeconfig /root/virtualcluster/vc-1.kubeconfig logs test-deploy-5f4bcd8c-nfvq4
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log test-deploy-5f4bcd8c-nfvq4))
➜ .minikube kubectl --kubeconfig /root/virtualcluster/vc-1.kubeconfig logs test-deploy-5f4bcd8c-nfvq4
Error from server (Forbidden): Forbidden (user=minikubeCA, verb=get, resource=nodes, subresource=proxy) ( pods/log test-deploy-5f4bcd8c-nfvq4)
➜ .minikube kubectl --kubeconfig /root/virtualcluster/vc-1.kubeconfig logs test-deploy-5f4bcd8c-nfvq4
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log test-deploy-5f4bcd8c-nfvq4))
So the cert/key in this dir is still the right one?
By the way, when I use the ocp env inteads of Kind or Minikube, how could I find the cert/key of ocp env?
Thanks!
Did you exactly follow the steps below to create the cert when using minikube?
Create kubelet client secret in minikube cluster
If you're using minikube, the client PKI files are located in ~/.minikube/. So we can create vc-kubelet-client secret using the following commands:
# Copy the files over
cp ~/.minikube/cert.pem client.crt && cp ~/.minikube/key.pem client.key
# Create a new secret
kubectl create secret generic vc-kubelet-client --from-file=client.crt --from-file=client.key --namespace vc-manager
Update vn-agent
To apply this secret to vn-agent Pod(s), one can patch the vn-agent DaemonSet to change the secret name of the kubelet-client-cert volume to the newly created vc-kubelet-client:
$ kubectl -n vc-manager patch daemonset/vn-agent --type json \
-p='[{"op": "replace", "path": "/spec/template/spec/volumes/0/secret/secretName", "value":"vc-kubelet-client"}]'
The vn-agent Pod(s) will be recreated in every node to talk with kubelet directly from now onwards.
For ocp env, I have not tried before so you'd ask your cluster administrator for the correct cert/key pairs.
It's weird that I follow this step by step exactly..
Anyway, I will try on the ocp env at the same time.
@Fei-Guo I missed mentioning that even removing the cert/key, the tenant cluster couldn't access the docker container/pod either. The same error info showed. And the log show the will forward request to super apiserver
➜ virtualcluster kubectl --kubeconfig /root/virtualcluster/vc-1.kubeconfig logs test-deploy-5f4bcd8c-nfvq4
Error from server (NotFound): the server could not find the requested resource ( pods/log test-deploy-5f4bcd8c-nfvq4)
➜ virtualcluster kubectl --kubeconfig /root/virtualcluster/vc-1.kubeconfig exec -it test-deploy-5f4bcd8c-nfvq4 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Error from server (NotFound): the server could not find the requested resource
➜ virtualcluster kubectl --context meta logs vn-agent-lz45k -n vc-manager
I0413 08:26:09.560989 1 cert.go:61] Using self-signed cert (vn.crt, vn.key)
I0413 08:26:09.561049 1 server.go:123] server listen on :10550
I0413 08:26:13.077942 1 route.go:124] request /containerLogs/default/test-deploy-5f4bcd8c-nfvq4/poc
I0413 08:26:13.078035 1 route.go:150] will forward request to super apiserver
I0413 08:26:13.078424 1 route.go:182] request after translate https://10.96.0.1:443
I0413 08:26:41.453019 1 route.go:124] request /containerLogs/default/test-deploy-5f4bcd8c-nfvq4/poc
I0413 08:26:41.453061 1 route.go:150] will forward request to super apiserver
I0413 08:26:41.453395 1 route.go:182] request after translate https://10.96.0.1:443
I0413 08:27:30.514197 1 route.go:124] request /exec/default/test-deploy-5f4bcd8c-nfvq4/poc?command=sh&input=1&output=1&tty=1
I0413 08:27:30.514281 1 route.go:150] will forward request to super apiserver
I0413 08:27:30.514711 1 route.go:182] request after translate https://10.96.0.1:443
Btw, did you build the vn-agent locally or use the image from the docker hub? The docker hub image was outdated and I just uploaded the latest one. I just verified that the latest version at least works in my local setup.
Sorry that we don't have a release yet but it will come soon in kubernetes-sigs/cluster-api-provider-nested repo.
The new image worked, thanks!