master cluster add a vkubelet sub cluster ，and secret use the master cluster kubeconfig ca key ？

[Amy Huang]

Describe the bug
master cluster add a vkubelet sub cluster ，and secret use the master cluster kubeconfig ca key ？

To Reproduce
Steps to reproduce the behavior:
1 master cluster config secret use master cluster kubeconfig ---->virtual-kubelet;
2 master cluster config configmap use sub cluster kubeconfig----->vk-infra-test;
3 deploy vkubelet in master cluster

apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-infra-test
namespace: kube-system
labels:
k8s-app: kubelet
spec:
replicas: 1
selector:
matchLabels:
k8s-app: virtual-kubelet
template:
metadata:
labels:
pod-type: virtual-kubelet
k8s-app: virtual-kubelet
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: pod-type
operator: In
values:
- virtual-kubelet
topologyKey: kubernetes.io/hostname
tolerations:
- effect: NoSchedule
key: role
value: not-vk
operator: Equal
hostNetwork: true
containers:
- name: virtual-kubelet
image: XXXXXX/docker/virtual-node:v0.1.1-30-g2cc1d681db013d
imagePullPolicy: IfNotPresent
env:
- name: KUBELET_PORT
value: "10450"
- name: APISERVER_CERT_LOCATION
value: /etc/virtual-kubelet/cert/cert.pem
- name: APISERVER_KEY_LOCATION
value: /etc/virtual-kubelet/cert/key.pem
- name: APISERVER_CA_CERT_LOCATION
value: /etc/virtual-kubelet/cert/ca.pem
- name: DEFAULT_NODE_NAME
value: virtual-kubelet
- name: VKUBELET_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: credentials
mountPath: "/etc/virtual-kubelet/cert"
readOnly: true
- name: kube
mountPath: "/root"
readOnly: true
args:
- --provider=k8s
- --nodename=k8s-infra-test
- --disable-taint=true
- --kube-api-qps=500
- --kube-api-burst=1000
- --client-qps=500
- --client-burst=1000
- --client-kubeconfig=/root/kube.config
- --klog.v=5
- --log-level=debug
- --metrics-addr=:10455
livenessProbe:
tcpSocket:
port: 10455
initialDelaySeconds: 20
periodSeconds: 20
volumes:
- name: credentials
secret:
secretName: virtual-kubelet
- name: kube
configMap:
name: vk-infra-test
items:
- key: kube.config
path: kube.config
defaultMode: 420
serviceAccountName: virtual-kubelet

Expected behavior

1 add vkubelet node success ，
2 then delete and add vkubelet again ，
the master cluster webhook MutatingWebhookConfiguration ‘s caBundle
x509 error

k8s_api_server
container_name:kube-apiserver
source:stderr
time:2023-07-13T01:48:43.95556069Z
content:W0713 09:48:43.955510 1 dispatcher.go:154] Failed calling webhook, failing closed mesh.XX.com: failed calling webhook "mesh.XX.com": failed to call webhook: Post "https://XX-mesh-webhook-svc.XX-mesh.svc:443/validate?timeout=10s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Kubernetes")

Kubernetes version (please complete the following information):

• Version of Upper K8s cluster
  1.24.6
• Version of Lower K8s cluster
  1.26.3

Additional context
Add any other context about the problem here.

[github-actions]

Thank your for opening your first issue :)

[Weidong Cai]

I am not sure I get it. Do you mean the tensile-kube webhook can not work or the webhook outside? This looks like a CA isuue.
If your service is running in the sub cluster and you want access then from the subcluster. I would suggest to use self-singed CA not the CA same as the kubernetes.