pmem / kvdk

Key Value Development Kit

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for supporting DRAM, Optane persistent memory and CXL memory pool. It also demonstrates several optimization methods for high performance with tiered memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like read-modify-write, checkpoint, etc.

Features

• Rich data types
  • string, sorted, hash, list, hash
• Basic KV operations
  • get/put/update/delete/scan
• Read-Modify-Write
• Support TTL
• Atomic Batch Write
• Snapshot based Scan
• Consistent Dump & Restore to/from storage
• Consistent Checkpoint
• Transaction
• C/C++/Java APIs

Limitations

• The maximum supported key-value size is 64KB-4GB according to configs.
• No approach to key-value compression.
• Users can't expand the persistent memory space on the fly.

Getting the Source

git clone --recurse-submodules https://github.com/pmem/kvdk.git

Building

Install dependent tools and libraries on Ubuntu 18.04

sudo apt install make clang-format-9 pkg-config g++ autoconf libtool asciidoctor libkmod-dev libudev-dev uuid-dev libjson-c-dev libkeyutils-dev pandoc libhwloc-dev libgflags-dev libtext-diff-perl bash-completion systemd wget git curl

git clone https://github.com/pmem/ndctl.git
cd ndctl
git checkout v70.1
./autogen.sh
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
make
sudo make install

git clone https://github.com/pmem/pmdk.git
cd pmdk
git checkout 1.11.1
make
sudo make install

wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
tar vzxf cmake-3.12.4.tar.gz
cd cmake-3.12.4
./bootstrap
make
sudo make install

Install dependent tools and libraries on CentOS 8

yum config-manager --add-repo /etc/yum.repos.d/CentOS-Linux-PowerTools.repo
yum config-manager --set-enabled PowerTools

yum install -y git gcc gcc-c++ autoconf automake asciidoc bash-completion xmlto libtool pkgconfig glib2 glib2-devel libfabric libfabric-devel doxygen graphviz pandoc ncurses kmod kmod-devel libudev-devel libuuid-devel json-c-devel keyutils-libs-devel gem make cmake libarchive clang-tools-extra hwloc-devel perl-Text-Diff gflags-devel curl

git clone https://github.com/pmem/ndctl.git
cd ndctl
git checkout v70.1
./autogen.sh
./configure CFLAGS='-g -O2' --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib
make
sudo make install

git clone https://github.com/pmem/pmdk.git
cd pmdk
git checkout 1.11.1
make
sudo make install

wget https://github.com/Kitware/CMake/releases/download/v3.12.4/cmake-3.12.4.tar.gz
tar vzxf cmake-3.12.4.tar.gz
cd cmake-3.12.4
./bootstrap
make
sudo make install

Compile KVDK

mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCHECK_CPP_STYLE=ON && make -j

How to test it on a system without PMEM

# set the correct path for pmdk library
export LD_LIBRARY_PATH=/usr/local/lib64

# setup a tmpfs for test
mkdir /mnt/pmem0
mount -t tmpfs -o size=2G tmpfs /mnt/pmem0

# force the program work on non-pmem directory
export PMEM_IS_PMEM_FORCE=1

cd kvdk/build/examples
# Note: this requires CPU supporting AVX512
./cpp_api_tutorial

Benchmarks

Here are the examples of how to benchmark the performance of KVDK on your systems.

Documentations

User Guide

Please refer to User guide for API introductions of KVDK.

Architecture

Support

Welcome to join the wechat group or slack channel for KVDK tech discussion.

• Wechat
• Slack Channel