matrixorigin / matrixone

Hyperconverged cloud-edge native database

Home Page:https://docs.matrixorigin.cn/en

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Bug]: 1.2-dev standalone regression: sysbench mo oom

heni02 opened this issue · comments

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

1.2-dev

Commit ID

efe7202

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9345673782/job/25719137036
企业微信截图_45793635-448b-4d97-9321-daa4b1ed5dfc
企业微信截图_6dbecf41-22f7-43c0-91d2-fe0b5cbbc7f4

profile 被清理掉了,今日复现补充profile

Expected Behavior

No response

Steps to Reproduce

sysbench  --mysql-host=127.0.0.1 --mysql-port=6001 --mysql-user=dump --mysql-*** select_random_points.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=30 --report-interval=10    prepare
sysbench  --mysql-host=127.0.0.1 --mysql-port=6001 --mysql-user=dump --mysql-***   select_random_points.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=300 --report-interval=10   --range_selects=off --point_selects=1 run

Additional information

related fix task:

@ouyuanning 麻烦看一下

单机Sysbench OOM分析

任务链接:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9364954572/job/25779210925

MO分支:1.2-dev

关键配置:CN MEMCACHE 32G;DN MEMCACHE 1G

heap profile:
heap.tar.gz

系统OOM时间点:19点18分38秒

image

全流程mo进程内存占用趋势图
memory_usage_over_time_with_smaller_labels

一些diff

Case1: OLTP Delete Test With 100 Treads

18:26 vs 18:29

alloc diff
image
inuse diff
image

Case2: OLTP Read Only Test With 100 Treads

Part1: create index 19:14 vs 19:15
alloc diff
image
inuse diff
image
Part2: read only 19:16 vs 19:17(已报错)
alloc diff
image
inuse diff
image

OOM前的gc日志

image

还要再分析