集群联邦join问题
ZhangChengJi opened this issue · comments
CLUSTER节点
[root@k8e-test1 ~]# kubectl get node -A
NAME STATUS ROLES AGE VERSION
k8e-test1 Ready control-plane,master 17m v1.21.10+k8e1a
master1节点
[root@h1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
h1.taosdata.com Ready control-plane,master 13m v1.21.10+k8e1a
为啥不显示对方node节点啊🤔
加节点和当master节点不太一样。master节点默认内置一个agent，标签不会打。下面就是加agent节点的例子
sudo cat <<EOF >> /etc/systemd/system/k8e.service.env
K8E_TOKEN=ilovek8e
K8E_NODE_NAME=k8e-test4
K8E_URL=https://172.25.1.56:6443
EOF
curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=ilovek8e K8E_URL=https://172.25.1.56:6443 sh -
`[root@h2 ~]# curl -sfL https://gitee.com/zhangchengji/yctest/raw/master/install.sh | K8E_TOKEN=ilovek8e K8E_URL=https://192.168.0.70:6443 sh -
Finding latest version from GitHub
v1.21.10+k8e1a
Downloading package https://gh.api.99988866.xyz/https://github.com/xiaods/k8e/releases/download/v1.21.10+k8e1a/k8e as /tmp/k8e
Download complete.
[INFO] Skipping /usr/local/bin/kubectl symlink to k8e, already exists
[INFO] Skipping /usr/local/bin/crictl symlink to k8e, already exists
[INFO] Skipping /usr/local/bin/ctr symlink to k8e, command exists in PATH at /usr/bin/ctr
[INFO] Create nerdctl symlink for k8e
[INFO] Create cilium ctl symlink for k8e
export CONTAINERD_ADDRESS=/run/k8e/containerd/containerd.sock
export PATH=$PATH:/usr/local/bin
alias docker=nerdctl
Loaded image: rancher/mirrored-metrics-server:v0.5.2
Loaded image: quay.io/cilium/cilium:v1.10.5
Loaded image: rancher/mirrored-coredns-coredns:1.8.6
Loaded image: rancher/mirrored-library-busybox:1.34.1
Loaded image: rancher/local-path-provisioner:v0.0.21
Loaded image: rancher/mirrored-library-traefik:2.5.6
Loaded image: rancher/mirrored-pause:3.5
Loaded image: quay.io/cilium/operator-generic:v1.10.5
Loaded image: rancher/klipper-helm:v0.6.6-build20211022
Loaded image: rancher/klipper-lb:v0.3.4
[INFO] Creating killall script /usr/local/bin/k8e-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k8e-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k8e.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k8e.service
[INFO] systemd: Enabling k8e unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k8e.service to /etc/systemd/system/k8e.service.
[INFO] systemd: Starting k8e
Verifying binaries in /var/lib/k8e/data/908d81bdaafdb96c287e3f09100ec40d4af66c04f619f367feef857526394e8b/bin:
- sha256sum: good
- links: good
System:
- /usr/sbin iptables v1.4.21: older than v1.8
- swap: should be disabled
- routes: ok
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
modprobe: FATAL: Module configs not found.
info: reading kernel config from /boot/config-5.16.13-1.el7.elrepo.x86_64 ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
Optional Features:
- CONFIG_USER_NS: enabled
(RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line) (fail) - CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
- "overlay":
- CONFIG_VXLAN: enabled (as module)
Optional (for encrypted networks):- CONFIG_CRYPTO: enabled
- CONFIG_CRYPTO_AEAD: enabled
- CONFIG_CRYPTO_GCM: enabled
- CONFIG_CRYPTO_SEQIV: enabled
- CONFIG_CRYPTO_GHASH: enabled
- CONFIG_XFRM: enabled
- CONFIG_XFRM_USER: enabled
- CONFIG_XFRM_ALGO: enabled
- CONFIG_INET_ESP: enabled (as module)
- CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- CONFIG_VXLAN: enabled (as module)
- "overlay":
- Storage Drivers:
- "overlay":
- CONFIG_OVERLAY_FS: enabled (as module)
- "overlay":
STATUS: 1 (fail)
[root@h2 ~]# nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE
[root@h2 ~]# cilium status
/¯¯
/¯¯_/¯¯\ Cilium: 1 errors
_/¯¯_/ Operator: 1 errors
/¯¯_/¯¯\ Hubble: 1 warnings
_/¯¯_/ ClusterMesh: 1 warnings
__/
Cluster Pods: 0/0 managed by Cilium
Errors: cilium cilium Get "http://localhost:8080/apis/apps/v1/namespaces/kube-system/daemonsets/cilium": dial tcp [::1]:8080: connect: connection refused
cilium-operator cilium-operator Get "http://localhost:8080/apis/apps/v1/namespaces/kube-system/deployments/cilium-operator": dial tcp [::1]:8080: connect: connection refused
Warnings: hubble-relay hubble-relay hubble relay is not deployed
hubble-ui hubble-ui hubble ui is not deployed
clustermesh-apiserver clustermesh-apiserver clustermesh is not deployed
[root@h2 ~]# cilium install
ℹ️ using Cilium version "v1.10.5"
🔮 Auto-detected IPAM mode: cluster-pool
❌ Cluster name "" is not valid, must match regular expression: ^a-z0-9$
↩️ Rolling back installation...
Error: Unable to install Cilium: invalid cluster name`
能帮我看看问题出在哪里吗🥺
1、k8e check-config是自动检测环境的命令，目前看环境中swap没有disable掉。
modprobe: FATAL: Module configs not found.2、systemctl status k8e看一下服务有没有启动成功，如果成功后使用journalctl -u k8e.service看一下日志
因为有错误，所以脚本退出了。k8e在启动时会自动解压出cilium安装工具cilium，所以你强行安装cilium前，需要提前声明一个kube config,
export KUBECONFIG=/etc/k8e/k8e.yaml
cilium install这样就可以安装了。
[root@h2 ~]# journalctl -u k8e.service
-- Logs begin at 五 2022-03-11 12:04:28 CST, end at 五 2022-03-11 13:48:05 CST. --
3月 11 12:14:45 h2.taosdata.com systemd[1]: [/etc/systemd/system/k8e.service:11] Failed to parse service type, ignoring: exec
3月 11 12:14:45 h2.taosdata.com systemd[1]: [/etc/systemd/system/k8e.service:11] Failed to parse service type, ignoring: exec
3月 11 12:14:45 h2.taosdata.com systemd[1]: [/etc/systemd/system/k8e.service:11] Failed to parse service type, ignoring: exec
3月 11 12:14:45 h2.taosdata.com systemd[1]: Starting Simple Kubernetes Distribution...
3月 11 12:14:45 h2.taosdata.com sh[6574]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
3月 11 12:14:45 h2.taosdata.com sh[6574]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
3月 11 12:14:45 h2.taosdata.com systemd[1]: Started Simple Kubernetes Distribution.
3月 11 12:14:45 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:45+08:00" level=info msg="Acquiring lock file /var/lib/k8e/data/.lock"
3月 11 12:14:45 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:45+08:00" level=info msg="Preparing data dir /var/lib/k8e/data/908d81bdaafdb96c287e3f09100ec40d4af66c04f619f367feef857526394e8b"
3月 11 12:14:48 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:48.858851559+08:00" level=info msg="Starting k8e agent v1.21.10+k8e1a (fea388b)"
3月 11 12:14:48 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:48.860159272+08:00" level=info msg="Running load balancer 127.0.0.1:6444 -> [192.168.0.70:6443]"
3月 11 12:14:48 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:48.945596636+08:00" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation
3月 11 12:14:48 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:48.954320592+08:00" level=error msg="Failed to configure agent: failed to retrieve configuration from server: https://127.0.0.1:6444/v1-k8e/config: 401 Unauthorized"
3月 11 12:14:53 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:53.964794697+08:00" level=error msg="Failed to configure agent: failed to retrieve configuration from server: https://127.0.0.1:6444/v1-k8e/config: 401 Unauthorized"
3月 11 12:14:58 h2.taosdata.com k8e[6584]: time="2022-03-11T12:14:58.974420693+08:00" level=error msg="Failed to configure agent: failed to retrieve configuration from server: https://127.0.0.1:6444/v1-k8e/config: 401 Unauthorized"
我现在把swap关闭了
这个问题就比较清晰了
`[root@h2 ~]# cat /etc/systemd/system/k8e.service
[Unit]
Description=Simple Kubernetes Distribution
Documentation=https://getk8e.com
After=network-online.target
Wants=network-online.target
[Install]
WantedBy=multi-user.target
[Service]
Type=exec
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k8e.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k8e \
agent \
[root@h2 ~]# cd /usr/local/bin/
[root@h2 bin]# ls
cilium crictl k8e k8e-killall.sh k8e-uninstall.sh kubectl nerdctl`
[root@h2 bin]# uname -a
Linux h2.taosdata.com 5.16.13-1.el7.elrepo.x86_64 #1 SMP PREEMPT Tue Mar 8 08:32:26 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@h2 ~]# k8e check-config
Verifying binaries in /var/lib/k8e/data/908d81bdaafdb96c287e3f09100ec40d4af66c04f619f367feef857526394e8b/bin:
- sha256sum: good
- links: good
System:
- /usr/sbin iptables v1.4.21: older than v1.8
- swap: disabled
- routes: ok
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
modprobe: FATAL: Module configs not found.
info: reading kernel config from /boot/config-5.16.13-1.el7.elrepo.x86_64 ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
- "overlay":
- CONFIG_VXLAN: enabled (as module)
Optional (for encrypted networks):
- CONFIG_CRYPTO: enabled
- CONFIG_CRYPTO_AEAD: enabled
- CONFIG_CRYPTO_GCM: enabled
- CONFIG_CRYPTO_SEQIV: enabled
- CONFIG_CRYPTO_GHASH: enabled
- CONFIG_XFRM: enabled
- CONFIG_XFRM_USER: enabled
- CONFIG_XFRM_ALGO: enabled
- CONFIG_INET_ESP: enabled (as module)
- CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
- "overlay":
- CONFIG_OVERLAY_FS: enabled (as module)
STATUS: pass
[root@h2 ~]#
我知道原因了。你没有配token。我也犯过这个错误。按照我这个来：
在master机器上：
curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=ilovek8e sh -
在agent机器上：
curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=ilovek8e K8E_URL=https://172.31.4.108:6443 sh -
这个token必须一样，不然master机器上会自动生成一个token，所以agent加不进去。
注意下，我的文档中是要求你自己配serivce.env，是为了持久化变量，不然systemd一重启就没有了。
我把install.sh脚本多安装了几遍。发现有一个执行不稳定的地方在影响cilium。我把install.sh脚本改进了。你重新复制下就可以了
我把install.sh脚本多安装了几遍。发现有一个执行不稳定的地方在影响cilium。我把install.sh脚本改进了。你重新复制下就可以了
你上传install.sh脚本了吗
yes
我测试了好几遍，没啥大问题
[root@h3 ~]# systemctl status k8e.service
● k8e.service - Simple Kubernetes Distribution
Loaded: loaded (/etc/systemd/system/k8e.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since 五 2022-03-11 17:09:58 CST; 4s ago
Docs: https://getk8e.com
Process: 6124 ExecStart=/usr/local/bin/k8e server (code=exited, status=203/EXEC)
Process: 6121 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 6118 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 6114 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 6124 (code=exited, status=203/EXEC)
3月 11 17:09:58 h3.taosdata.com systemd[1]: Failed to start Simple Kubernetes Distribution.
3月 11 17:09:58 h3.taosdata.com systemd[1]: Unit k8e.service entered failed state.
3月 11 17:09:58 h3.taosdata.com systemd[1]: k8e.service failed.
直接😩起不来了
直接😩起不来了
卸载后重装一遍看看
sudo /usr/loca/bin/k8e-killall.sh
sudo /usr/local/bin/k8e-uninstall.sh
curl -sfL https://getk8e.com/install.sh | sh -试试，如果还是不行，方便提供下
journalctl -u k8e.service脚本是可以重复安装的
微信方便沟通一下吗wx : Zcjvsgola
Zcjvsgola
已经安排了
done