andrew2015 / dkv

Distributed KV data store with tunable consistency, synchronous replication

dkv

DKV is a distributed key value store server written in Go. It exposes all its functionality over gRPC & Protocol Buffers.

Features

• Data Sharding
• Tunable consistency
• Data replication over WANs

Supported APIs

• createVBucket(replicationFactor)
• put(K,V,vBucket)
• del(K,vBucket)
• get(K,consistency)

Design

Dependencies

• Go version 1.16+
• RocksDB v6.22.1 as a storage engine
• GoRocksDB provides the CGo bindings with RocksDB
• Badger v1.6 as a storage engine
• Nexus for sync replication over Raft consensus

DKV on Docker

Follow these instructions to launch a DKV container using the Dockerfile included.

$ curl -fsSL https://raw.githubusercontent.com/flipkart-incubator/dkv/master/Dockerfile | docker build -t dkv/dkv-deb9-amd64 -f - .
$ docker run -it dkv/dkv-deb9-amd64:latest dkvsrv --help

Building DKV on Mac OSX

Installing Dependencies

DKV depends on RocksDB, and its CGo bindings, so we need to install rocksdb along with its dependecies.

• Ensure HomeBrew is installed
• brew install rocksdb zstd

Building DKV

$ mkdir -p ${GOPATH}/src/github.com/flipkart-incubator
$ cd ${GOPATH}/src/github.com/flipkart-incubator
$ git clone https://github.com/flipkart-incubator/dkv
$ cd dkv
$ make build

If you want to build for other platform, set GOOS, GOARCH environment variables. For example, build on macOS for linux like following:

$ make GOOS=linux build

Running

Once DKV is built, the <PROJECT_ROOT>/bin folder should contain the following binaries:

• dkvsrv - DKV server program
• dkvctl - DKV client program

Launching the DKV server in standalone mode

A single DKV instance can be launched using the following command:

$ ./bin/dkvsrv \
    -db-folder <folder_name> \
    -listen-addr <host:port> \
    -db-engine <rocksdb|badger>

$ ./bin/dkvctl -dkvAddr <host:port> -set <key> <value>
$ ./bin/dkvctl -dkvAddr <host:port> -get <key>

Example session:

$ ./bin/dkvsrv -db-folder /tmp/db -listen-addr 127.0.0.1:8080 -db-engine rocksdb
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -set foo bar
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -get foo
bar
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -set hello world
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -get hello
world
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -del foo
$ ./bin/dkvctl -dkvAddr 127.0.0.1:8080 -iter "*"
hello => world

Launching the DKV server for synchronous/asynchronous replication

Please refer to the wiki instructions on how to run DKV in cluster mode.

Documentation

Detailed documentation on specific features, design principles, data guarantees etc. can be found in the dkv Wiki

Testing

If you want to execute tests inside DKV, run this command:

$ make test

Packaging

Linux

$ make GOOS=linux dist

macOS

$ make GOOS=darwin dist

Support

dkv is undergoing active development. Consider joining the dkv-interest Google group for updates, design discussions, roadmap etc. in the initial stages of this project.