[Bug]: [date 6.6]tke regression: tpcc 500ware 1000threads test cn oom

[heni02]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

288fcef

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9402783044/job/25908076699

mo log:
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22hXV%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240606%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221717706820000%22,%22to%22:%221717707057000%22%7D%7D%7D&schemaVersion=1&orgId=1

profile:
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22dnH%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240606%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22memory:alloc_objects:count:space:bytes%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221717706817493%22,%22to%22:%221717706914132%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke regression tpcc 500ware 1000threads test

Additional information

No response

[heni02]

mo 自带的profile信息，抓取的两个时间2024-06-07 04:46:34 2024-06-07 04:47:03，oom前和oom后
hn_download_tmp.zip

[heni02]

date 6.7 regression也复现了 cc @reusee
job：
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9418697195/job/25953862790

profile：
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22oS6%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240607%5C%22,pod%3D%5C%22nightly-regression-dis-tp-cn-9c8hq%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22memory:alloc_objects:count:space:bytes%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221717789930000%22,%22to%22:%221717789990000%22%7D%7D%7D&schemaVersion=1&orgId=1

[reusee]

https://grafana.ci.matrixorigin.cn/d/c94fd2f4-b3b5-401f-b3d3-f998584c097a/go-runtime-metrics?orgId=1&var-interval=1m&var-namespace=mo-nightly-regression-20240610&var-pod=nightly-regression-dis-tp-cn-5x5j5&from=1718029800000&to=1718118000000&viewPanel=159

从这个指标看，是 cn 的 go 堆占用很多。在达到 54.2G 之后，发生了 OOM。
估计还是 mpool 分配的内存太多的问题。

[reusee]

https://grafana.ci.matrixorigin.cn/goto/noMbrI8SR?orgId=1

从 heap profile 看，也是 go 堆占用高。最高时到了 91.9GB。次高峰是 47.2GB。这个量加上堆外的内存，是超过限额的了。

后续的优化方向，是mpool的重构。

[reusee]

mpool重构中

[reusee]

大致可以确定是goroutine泄露，具体的泄露路径，需要pyroscope开启goroutine profile，才能看到。

[reusee]

继续优化

[reusee]

继续优化

[reusee]

已优化