VLDB Summer School 2021 Labs

Introduction

This is the labs of VLDB Summer School 2021. The target is to build a distributed database.

There are several modules in a distributed database.

TinyKV, the storage engine of the system.
TinyScheduler, it is used to manager and schedule TinyKV cluster.
TinySQL, the SQL layer of TinyKV engine.

Labs

There are 4 labs in this course.

Lab 1, implement the storage and log layer in TinyKV.
Lab 2, implement the transaction layer in TinyKV.
Lab 3, implement the Percolator protocol.
Lab 4, implement the SQL execution layer.
- Lab 4-A, implement SQL protocol.
- Lab 4-B, implement update executor.
- Lab 4-C, implement select and projection executor.

The code is separated into 2 parts, TinyKV and TinyScheduler is in tinykv, and TinySQL is in tinysql.

You need to follow the order in the labs chapter. You may learn more from the README files in TinyKV and TinySQL.

Autograding

The details of classroom usage can be found in the classroom doc.

Autograding is a workflow which can automatically run test cases. However there are some limitations in Github classroom, in order to make golang works and run it in our self-hosted machines, you need to overwrite the workflow generated by Github classroom and commit it.

cp scripts/classroom.yml .github/workflows/classroom.yml

If you don't use GitHub classroom, just fork this repo, work in your repo, test locally and send a email with your repository address to us after complete some or all the tasks.

Getting started

First, please clone the repository with git to get the source code of the project.

git clone https://github.com/vldbss-2021/vldb-2021-labs-{{username}}.git

Then make sure you have installed go >= 1.13 toolchains. You should also have installed make. Now you can run make under tinykv or tinysql dir to check that everything is working as expected. You should see it runs successfully.

Deploy a cluster

Rather than a course, you can try TinyKV by deploying a real cluster, and interact with it through TinySQL.

Build

cd tinykv
make kv

It builds the binary of tinykv-server and tinyscheduler-server to bin dir.

cd tinysql
make server

It buillds the binary of tinysql-server to bin dir.

Deploy By Hand

Put the binary of tinyscheduler-server, tinykv-server and tinysql-server into a single dir. Under the binary dir, run the following commands:

mkdir -p data

./tinyscheduler-server

./tinykv-server -path=data

./tinysql-server --store=tikv --path="127.0.0.1:2379"

Deploy Use Cluster Command

Deploy the cluster in the local environment

# compile the cluster binary
cd tinykv
make deploy-cluster

# deploy the cluster, by default the number of scheduler server is 1 and the number of kv server is 3.
./bin/cluster deploy

# start the deployed cluster
./bin/cluster start

# stop the deployed cluster
./bin/cluster stop

# update the binary, please stop the cluster then do the upgrade
./bin/cluster upgrade

# unsafe destroy the whole cluster
./bin/cluster destroy

Note this does not deploy a tinysql server, to deploy a tinysql server, use

./tinysql-server --store=tikv --path="127.0.0.1:2379"

Play

mysql -u root -h 127.0.0.1 -P 4000

skyzh / vldb-2021-labs