bench tool: Set error happened when using sorted-type and latency enable

[Lin,Xiang]

"Set error" happened when using sorted-type and latency enable, option type inclues "Fill, update, Insert"，for example：

Insert new sorted-type kv
Write latency overflow: 12191390 us
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
sh: line 1: 71030 Segmentation fault (core dumped) numactl --cpunodebind=0 --membind=0 ./bench -latency=1 -populate=1 -value_size=128 -threads=96 -time=10 -path=/mnt/pmem0/kvdk -num=789516047 -space=536870912000 -max_write_threads=96 -fill=0 -type=sorted -read_ratio
=0 -existing_keys_ratio=0 > ./results/sorted_vs128_insert_thread96

[peifeng si]

Hi Sean,

Can you provide the command for above error? I can't reproduce it with:
numactl --cpunodebind=0 --membind=0 ./bench -latency=1 -populate=1 -value_size=128 -threads=96 -time=10 -path=/mnt/pmem0/kvdk -num=789516047 -space=136870912000 -max_write_threads=96 -fill=0 -type=sorted -read_ratio=0 -existing_keys_ratio=0
[LOG] time 0 ms: Initializing PMEM size 136870912000 in file /mnt/pmem0/kvdk/data
[LOG] time 16371 ms: Map pmem space done
[LOG] time 17203 ms: In restoring: iterated 0 records
[LOG] time 17203 ms: Populating PMEM space ...
[LOG] time 83252 ms: Populating done
calculate latencies
init 96 write threads
init 0 read threads
------- ops in seconds -----------
time (ms), read ops, not found, write ops, total read, total write
1000 0 0 423000 0 423000
2000 0 0 531000 0 954000
3000 0 0 576000 0 1530000
4000 0 0 581000 0 2111000
5000 0 0 670000 0 2781000
6000 0 0 711000 0 3492000
7000 0 0 803000 0 4295000
8000 0 0 864000 0 5159000
9000 0 0 851000 0 6010000
10000 0 0 1042000 0 7052000
finish bench
------------ statistics ------------
read ops 0, write ops 705200
write lantencies (us): Avg: 136.54, P50: 22.90, P99: 794.70, P99.5: 960.10, P99.9: 1168.60, P99.99: 1260.10
[LOG] time 122288 ms: instance closed

[Lin,Xiang]

reproduced this issue, happened at using a lot of threads, seems related to multi-threaded write competition.

numactl --cpunodebind=0 --membind=0 ./bench -latency=1 -populate=1 -value_size=4096 -threads=96 -time=10 -path=
/mnt/pmem0/kvdk -num=26163299 -space=536870912000 -max_write_threads=96 -fill=0 -type=sorted -read_ratio=0 -exi
sting_keys_ratio=0 > ./results/sorted_vs4096_insert_thread96
Write latency overflow: 16774060 us
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error
Set error

[Jiayu Wu]

I think this is because it run out of space which cause a set error, and the write thread need take long time to seek free space which causes a large latency, and make a segment fault in bench.cpp.

I will resolve this later. Thank you.