在通过daemonset方式启动kmesh时kubectl logs报错

Question

在通过daemonset方式启动kmesh时kubectl logs报错

lhy9816 opened this issue 7 months ago · comments

openeuler23-03环境，在安装istio-1.19.3之后通过kubectl apply -f kmesh.yaml安装kmesh时候报错，该 kmesh-deploy daemonset可以在k8s集群中运行

但是kubectl logs kmesh-deploy-t4zcj 会出现如下报错

[root@host-192-168-100-230 docker]# kubectl logs kmesh-deploy-t4zcj
time="2023-12-08T08:39:01Z" level=info msg="options InitDaemonConfig successful" subsys=manager
time="2023-12-08T08:39:01Z" level=info msg="bpf Start successful" subsys=manager
time="2023-12-08T08:39:01Z" level=info msg="controller Start successful" subsys=manager
time="2023-12-08T08:39:01Z" level=info msg="command StartServer successful" subsys=manager
panic: runtime error: index out of range [3] with length 0

goroutine 26 [running]:
encoding/binary.littleEndian.Uint32(...)
        /usr/lib/golang/src/encoding/binary/binary.go:80
oncn.io/mesh/pkg/nets.ConvertIpToUint32({0xc00044f8f0?, 0xc0004461b0?})
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/nets/nets.go:34 +0x4e
oncn.io/mesh/pkg/controller/envoy.newApiSocketAddress(0xb?)
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_loader.go:138 +0xf1
oncn.io/mesh/pkg/controller/envoy.newApiClusterLoadAssignment(0xc00047d1a0)
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_loader.go:107 +0x2ad
oncn.io/mesh/pkg/controller/envoy.(*AdsLoader).CreateApiClusterByCds(0xc000500f60, 0x2, 0xc00055f000)
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_loader.go:76 +0x258
oncn.io/mesh/pkg/controller/envoy.(*ServiceEvent).handleCdsResponse(0xc0005a1860, 0xc0000d4d80)
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_event.go:150 +0x393
oncn.io/mesh/pkg/controller/envoy.(*ServiceEvent).processAdsResponse(0xc0005a1860, 0xc0000d4d80)
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_event.go:112 +0x1a9
oncn.io/mesh/pkg/controller/envoy.(*AdsClient).runControlPlane(0xc000430690, {0x1b26a90, 0xc0008cf300})
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_client.go:142 +0x15c
created by oncn.io/mesh/pkg/controller/envoy.(*AdsClient).Run
        /root/kmesh/test/mugen-master/testcases/smoke-test/kmesh/oe_test_service_function/rpmbuild/BUILD/kmesh-0.0.1/pkg/controller/envoy/ads_client.go:160 +0x125

尝试demo示例也没有看到kmesh参与流量治理

leb-bit · Answer 1 · Mon Dec 11 2023 09:16:57 GMT+0800 (China Standard Time)

该问题是由于环境上istiod所下发xds配置无法正确解析，导致kmesh解析出来的字段中部分ip段为空，在go控制面
/usr/lib/golang/src/encoding/binary/binary.go:80中进行IP的大小端转换的时候对内存进行检查发现IP为空，导致panic

func (littleEndian) Uint32(b []byte) uint32 {
_ = b[3] // bounds check hint to compiler; see golang.org/issue/14808
return uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24
}

可能是kmesh的七层对istio 1.19版本的支持还不够完善，在较低版本(istio<=1.18.3 k8s <=1.20)支持良好，后续会测试并完善更多版本支持

Hangyu Li · Answer 2 · Thu Dec 14 2023 11:43:05 GMT+0800 (China Standard Time)

好的，确实是istio版本的问题，我降到1.14.5就可以了。