Temporary drop of insert performance
pashkinelfe opened this issue · comments
Reproduction same as for #333 run at be17f3a
after ~150 s TPS drops down to 0 then normalizes.
progress: 50.0 s, 28315.0 tps, lat 1.764 ms stddev 0.998, 0 failed
progress: 60.0 s, 27997.0 tps, lat 1.784 ms stddev 0.930, 0 failed
progress: 70.0 s, 28904.3 tps, lat 1.728 ms stddev 0.884, 0 failed
progress: 80.0 s, 27685.7 tps, lat 1.804 ms stddev 1.139, 0 failed
progress: 90.0 s, 26667.0 tps, lat 1.873 ms stddev 1.077, 0 failed
progress: 100.0 s, 29886.1 tps, lat 1.671 ms stddev 1.166, 0 failed
progress: 110.0 s, 25464.7 tps, lat 1.687 ms stddev 1.029, 0 failed
progress: 120.0 s, 4163.7 tps, lat 1.685 ms stddev 0.463, 0 failed
progress: 130.0 s, 133.0 tps, lat 1.308 ms stddev 0.504, 0 failed
progress: 140.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 150.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 160.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 170.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 180.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 190.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 200.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 210.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 220.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 230.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 240.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 250.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 260.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 270.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 280.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 290.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 300.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 310.0 s, 2575.0 tps, lat 361.894 ms stddev 8346.032, 0 failed
progress: 320.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 330.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 340.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 350.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 360.0 s, 26920.9 tps, lat 11.824 ms stddev 916.268, 0 failed
progress: 370.0 s, 27298.0 tps, lat 1.830 ms stddev 15.994, 0 failed
progress: 380.0 s, 26496.1 tps, lat 1.885 ms stddev 15.847, 0 failed
progress: 390.0 s, 28311.0 tps, lat 1.764 ms stddev 0.997, 0 failed
progress: 400.0 s, 29705.0 tps, lat 1.682 ms stddev 0.915, 0 failed
progress: 410.0 s, 30454.0 tps, lat 1.640 ms stddev 0.822, 0 failed
progress: 420.0 s, 31545.9 tps, lat 1.583 ms stddev 0.964, 0 failed
In the log putting map files to s3
2024-01-09 16:11:26.527 UTC [1614001] DEBUG: S3 part put orioledb_data/1/5/33802.0.0 orioledb_data/5/33802-1
2024-01-09 16:11:26.530 UTC [1613977] DEBUG: S3 map put orioledb_data/1/5/33802.map orioledb_data/5/33802-1.map
2024-01-09 16:11:26.536 UTC [1613924] DEBUG: S3 part put orioledb_data/1/5/33808.0.0 orioledb_data/5/33808-1
2024-01-09 16:11:26.539 UTC [1613824] DEBUG: S3 map put orioledb_data/1/5/33808.map orioledb_data/5/33808-1.map
2024-01-09 16:11:26.543 UTC [1614039] DEBUG: S3 part put orioledb_data/1/5/33814.0.0 orioledb_data/5/33814-1
2024-01-09 16:11:26.545 UTC [1614019] DEBUG: S3 map put orioledb_data/1/5/33814.map orioledb_data/5/33814-1.map
2024-01-09 16:11:26.549 UTC [1630657] DEBUG: S3 part put orioledb_data/1/5/33820.0.0 orioledb_data/5/33820-1
2024-01-09 16:11:26.566 UTC [1614018] DEBUG: S3 map put orioledb_data/1/5/33820.map orioledb_data/5/33820-1.map
2024-01-09 16:11:26.567 UTC [1613968] DEBUG: S3 part put orioledb_data/1/5/33826.0.0 orioledb_data/5/33826-1
2024-01-09 16:11:26.574 UTC [1613984] DEBUG: S3 map put orioledb_data/1/5/33826.map orioledb_data/5/33826-1.map
2024-01-09 16:11:26.574 UTC [1614003] DEBUG: S3 part put orioledb_data/1/5/33832.0.0 orioledb_data/5/33832-1
2024-01-09 16:11:26.574 UTC [1613818] DEBUG: S3 map put orioledb_data/1/5/33832.map orioledb_data/5/33832-1.map
Bgwriter takes 100% CPU. Wallclock profiler flame graph 30s @ 1000samples/s
Non-recompressed svg:
bgw.svg.zip
No drop anymore at 02bc078