Performance drops dramatically if update with 16 bytes value on dataset filled with 120 bytes value.
peifengsi opened this issue · comments
Bug Report
KVDK version
commit 14a9aa9
Date: Wed Dec 1 15:06:56 2021
System configuration
CPU: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
PMEM: 128 GB x 8
Reproduce steps
- fill data (value size = 120 bytes)
numactl --cpunodebind=1 --membind=1 ../build/bench -type=string -value_size=120 -threads=64 -max_write_threads=64 -time=10 -path=/mnt/pmem1/kvdk_string -num=805306368 -space=412316860416 -collections=16 -fill=1 -populate=1
- update (value size = 16 bytes)
numactl --cpunodebind=1 --membind=1 ../build/bench -type=string -value_size=16 -threads=64 -max_write_threads=64 -time=10000 -path=/mnt/pmem1/kvdk_string -num=805306368 -space=412316860416 -collections=16 -fill=0 -read_ratio=0 -key_distribution=random
[INFO] time 3894 ms: Map pmem space done
[INFO] time 4941 ms: RestorePendingBatch done: iterated 0 records
[INFO] time 11615 ms: RestoreData done: iterated 806092662 records
[INFO] time 11615 ms: Rebuild skiplist done
init 64 write threads
init 0 read threads
------- ops in seconds -----------
time (ms), read ops, not found, write ops, total read, total write
1000 0 0 38951000 0 38951000
2000 0 0 38459000 0 77410000
3000 0 0 39947000 0 117357000
4000 0 0 29682000 0 147039000
5000 0 0 2153000 0 149192000
6000 0 0 2128000 0 151320000
7000 0 0 2058000 0 153378000
8000 0 0 2053000 0 155431000
9001 0 0 2104000 0 157535000
10001 0 0 2002000 0 159537000
11001 0 0 1668000 0 161205000
12001 0 0 961000 0 162166000
13001 0 0 941000 0 163107000
14001 0 0 964000 0 164071000
15001 0 0 947000 0 165018000
16002 0 0 963000 0 165981000
17002 0 0 1314000 0 167295000
18002 0 0 1324000 0 168619000
19002 0 0 1303000 0 169922000
20002 0 0 1285000 0 171207000
21002 0 0 1306000 0 172513000
Expect behavior
write ops doesn't go down.
Current behavior
write ops slow down from 30Mops to 1Mops.
Debugging
Here is the hotspot got from vtune.
fixed in #137