[Bug]: 1.2-dev standalone regression: sysbench mo oom
heni02 opened this issue · comments
Is there an existing issue for the same bug?
- I have checked the existing issues.
Branch Name
1.2-dev
Commit ID
Actual Behavior
job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9345673782/job/25719137036
profile 被清理掉了,今日复现补充profile
Expected Behavior
No response
Steps to Reproduce
sysbench --mysql-host=127.0.0.1 --mysql-port=6001 --mysql-user=dump --mysql-*** select_random_points.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=30 --report-interval=10 prepare
sysbench --mysql-host=127.0.0.1 --mysql-port=6001 --mysql-user=dump --mysql-*** select_random_points.lua --mysql-db=sysbench_db --tables=10 --table_size=1000000 --threads=100 --time=300 --report-interval=10 --range_selects=off --point_selects=1 run
Additional information
related fix task:
- #16653
- #16655
- #16656
- #16659
- #16387
- https://github.com/matrixorigin/MO-Cloud/issues/3472
- https://github.com/matrixorigin/MO-Cloud/issues/3473
- https://github.com/matrixorigin/MO-Cloud/issues/3474
- https://github.com/matrixorigin/MO-Cloud/issues/3475
- https://github.com/matrixorigin/MO-Cloud/issues/3476
- https://github.com/matrixorigin/MO-Cloud/issues/3477
@ouyuanning 麻烦看一下
单机Sysbench OOM分析
任务链接:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9364954572/job/25779210925
MO分支:1.2-dev
关键配置:CN MEMCACHE 32G;DN MEMCACHE 1G
heap profile:
heap.tar.gz
系统OOM时间点:19点18分38秒
![image](https://private-user-images.githubusercontent.com/77312370/336479859-5e9b4623-58d8-42d1-b77d-d8b8f56a263e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY0Nzk4NTktNWU5YjQ2MjMtNThkOC00MmQxLWI3N2QtZDhiOGY1NmEyNjNlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ1ZWUzOTZjYjRiYzBhNTJmOTBlYTdlNjJlNDE2OTdhMGVhMWJiNTY0N2VlMzY4MmUxOGYxYmFlMDVmYWVkODYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.hZkVzaQhwXmiwPwWqyMINW6iGcfMly4lSeg8wWncW7w)
一些diff
Case1: OLTP Delete Test With 100 Treads
18:26 vs 18:29
alloc diff
![image](https://private-user-images.githubusercontent.com/77312370/336657340-5e2a4e7d-f5c3-4264-a4bf-439cfc9dc8d7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY2NTczNDAtNWUyYTRlN2QtZjVjMy00MjY0LWE0YmYtNDM5Y2ZjOWRjOGQ3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY4ZjliNzg4Y2EzMmFiYTY4NDY3OTkxNmM1YjA0OGFhNjBhMTMxZTNiODg1ZGM0MDUzNDA3ZmJjOTY5ZTJlMWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Q_Y9euPXrVSq2GzUNGPA-ahqrb-KfE_HGzdxBEyudkk)
inuse diff
![image](https://private-user-images.githubusercontent.com/77312370/336480016-4b261ad7-a470-4e88-800a-ae10fa5c7c0c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY0ODAwMTYtNGIyNjFhZDctYTQ3MC00ZTg4LTgwMGEtYWUxMGZhNWM3YzBjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThhYzY5NGU4NGQ5ZTM4NjJmNmFkMWZiZDk2ZTkwNGJiZmYyNjRkYmJkZmZmYjRiZDFhMGVlNzZmMDVlNjI0YWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.uLWryN4LYfaJBFMAorYkacKIY28aC0NiC56anXT6ZSk)
Case2: OLTP Read Only Test With 100 Treads
Part1: create index 19:14 vs 19:15
alloc diff
![image](https://private-user-images.githubusercontent.com/77312370/336658800-ba7b5aac-91a8-4e2d-82b8-0e4cce6014d9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY2NTg4MDAtYmE3YjVhYWMtOTFhOC00ZTJkLTgyYjgtMGU0Y2NlNjAxNGQ5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0OWYwZTY4NDg0ZDI0NmVlMjE3ZWE3MTJjZDk0ZGViM2UyNDkxNmYzYjE3OWFhZDViZDNkOWU1YTRmZTYwYmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.zFXskw-CJ7O1gip_g-4LgslEzFhKgYz6uoBCZSM6WR0)
inuse diff
![image](https://private-user-images.githubusercontent.com/77312370/336480067-82fca6b6-50ef-4316-a6d0-6bb8c4806c83.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY0ODAwNjctODJmY2E2YjYtNTBlZi00MzE2LWE2ZDAtNmJiOGM0ODA2YzgzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJjODVlYmMxYmFjN2RkMGQwZjViZjY2NzJmNTczMWVjMjIwYTM4ZTBlMThhZjhkMjg2NzA0YWNlOGI2Zjk0ZTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Ffis_rJ9WdHzjeCJHxVBOLxmTsqryA7wWSz86EgHGR4)
Part2: read only 19:16 vs 19:17(已报错)
alloc diff
![image](https://private-user-images.githubusercontent.com/77312370/336659015-58ffb65a-b089-4ab4-afb5-2dab3378dc02.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY2NTkwMTUtNThmZmI2NWEtYjA4OS00YWI0LWFmYjUtMmRhYjMzNzhkYzAyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI4MjA3NzdjYWVmMmE4NDc3NjI1MmQyNjFlNGM3OWE0YjdkZjg1MjViNmJhMWQzMzFjNzMxYmUzNTdlZDVlNzQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.M4ei5Bh3a5nU9qozwNZgtuNdffYOT1d6ZiTn5w1ErjQ)
inuse diff
![image](https://private-user-images.githubusercontent.com/77312370/336480128-9dafe5a6-3cb9-45ff-8b74-e24cc55b2434.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY0ODAxMjgtOWRhZmU1YTYtM2NiOS00NWZmLThiNzQtZTI0Y2M1NWIyNDM0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY4YTAyNjIzMWE1ZWYzM2JlM2ZmYzllODIxMjRhYWQ0ODQ4NjFmMWM5NjRjYjBjM2JjNjAxMzRjYmJjZjdhYmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.gbjPR6PdgNUDrmmI16Y34n2Rkq7hXLpnJ_PcGHIHnCs)
OOM前的gc日志
![image](https://private-user-images.githubusercontent.com/77312370/336659458-0431f5fd-cf54-424d-8ef7-55dd3950ff51.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjI3MjYzNDMsIm5iZiI6MTcyMjcyNjA0MywicGF0aCI6Ii83NzMxMjM3MC8zMzY2NTk0NTgtMDQzMWY1ZmQtY2Y1NC00MjRkLThlZjctNTVkZDM5NTBmZjUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODAzVDIzMDA0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA4ZjYxNGViZmIzYTExMDM2ZGY4YjliYjAyMjE4NWQ0YWMyNGJjNDhmOWY1ZTAzODNiNzg5MzZmZDg3NDBlZDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Of4bHDNaavH-1roLIrv15U0HR7HGVfO3VriVRdsxmhw)
还要再分析