wz30 / lightning

Lightning In-Memory Object Store

Lightning In-Memory Object Store

Lightning is a high-performance in-memory object store. Compared to traditional in-memory object stores (e.g., Redis, Memcached, Plasma), Lightning does not have inter-process communication (IPC) overheads on client operations (e.g., object creation, object fetching, object deletion).

Our VLDB 2022 paper (https://danyangzhuo.com/papers/VLDB22-Lightning.pdf) describes the technical details. Please email Danyang Zhuo ([e-mail]) if you have any question.

This is just a research prototype. Please don't use it in production systems.

Requirement

Lightning runs on the following configuration:

• Linux (4.15.0)
• Clang (6.0.0)
• Boost (1.65.1)
• Z3 (4.8.9)
• Mono (6.8.0)
• Dafny (2.3.0)

Docker

We suggest you use docker to construct the environment for compiling and running Lightning. We provide a dockerfile to simply this process.

First, you need to build the docker image.

docker build -t lightning .

Second, you need to instantiate a container. You need to enlarge the size of maximum shm the container use to at least 10G.

docker run -it --rm --shm-size=10g lightning

Build

mkdir build
cd build
cmake -DVERIFIER=ON ..
make -j

If you don't want to build the verifier, you can delete the "-DVERIFIER=ON" flag.

Run

./store

In another terminal,

./benchmark

Verify

We verify Lightning's crash fault isolation property in two steps.

Step #1: Verify the correctness of log implementation

dafny ../verifier/undo_log.dfy

Step #2: Verify that Lightning's C++ implementation uses the log correctly

./verifier/verify_num_logwrite

This will take around 10-15 minutes.

Build the Java Client

mkdir build
cd build
cmake -DJAVA_CLIENT=ON ..
make -j

Build the Python Client

cd python
make