[Bug]: [date 6.3]tke regression: tpcc 1000ware 1000threads reported TT_NEW_ORDER CONNECTION Communications link failure

[heni02]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

a37c7e5

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job：https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9352100991/job/25747773466

cn没有重启

mo log:
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22_0q%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240603%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221717444641572%22,%22to%22:%221717444841264%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke tpcc 1000ware 1000threads test

Additional information

No response

[XuPeng-SH]

https://grafana.ci.matrixorigin.cn/d/c4e17979-09c7-425e-8038-c33897e84a44/fileservice-metrics?orgId=1&var-interval=1m&var-namespace=mo-nightly-regression-20240603&var-pod=All&from=1717444581000&to=1717445121000

The s3-read latency is very high during that time and then the life of some transactions may exceed 60s, which is the timeout time of the TPCC client.

[YANGGMM]

@reusee 麻烦看一下

[reusee]

metrics里看到的时间，不只是读写s3 的时间，也包括调用端的处理时间
已经更新metrics，后面再进一步定位