benoitc/cbt

#cbt

multi-layer MVCC log append-only database library based on the Apache CouchDB btree.

Changes compared to couchdb

Pluggable Storage backends
use CRC32 to check data integrity instead of MD5
rewrote the file init part to make it more robust
removed the need of an external config. Provides correct default and use erlang environment.
documentation and specs
some syntax improvements.

build

1. install rebar

To build cbt you need to install rebar in your PATH. Rebar is available on Github:

https://github.com/rebar/rebar

Follow the README to install it.

2. build

Fetch the source code:

$ git clone https://bitbucket.org/refugeio/cbt.git

Build the source, run the make command. It will fetch any needed dependencies.

$ cd /<PATH_TO>/cbt
$ make

Note: To use snappy or lz4 compression methods you need to install the following dependencies:

snappy: https://github.com/fdmanana/snappy-erlang-nif

lz4: https://github.com/krestenkrab/erlang-lz4

And launch your application installed in the library path.

3. test CBT

Run the following command line:

$ make test

3. Build the doc

$ make doc

and open the index.html file in the doc folder. Or read it online.

Example of usage with the file backend

Example of usage:

Store a {Key Value} pair in a btree:

1> {ok, Fd} = cbt_file:open("test.db", [create_if_missing]).
{ok,<0.35.0>}
2> {ok, Btree} = cbt_btree:open(nil, Fd).
{ok,{btree,<0.35.0>,nil,undefined,undefined,undefined,nil,
           snappy,1279}}
3>
3> {ok, Btree2} = cbt_btree:add(Btree, [{a, 1}]).
{ok,{btree,<0.35.0>,
           {0,[],32},
           undefined,undefined,undefined,nil,snappy,1279}}
4> Root = cbt_btree:get_state(Btree2).
{0,[],32}
5> Header = {1, Root}.
{1,{0,[],32}}
6> cbt_file:write_header(Fd, Header).
ok

What we did here is to open a file, create a btree inside and add a key value. Until we write the header, the database value is not changed.

Now open the database in a new process and read the btree using the last header:

7> {ok, Fd1} = cbt_file:open("test.db").
{ok,<0.44.0>}
8>
8> {ok, Header1} = cbt_file:read_header(Fd1).
{ok,{1,{0,[],32}}}
9> Header1 == Header
9> .
true
10> {_, ReaderRoot} = Header1.
{1,{0,[],32}}
11> {ok, SnapshotBtree} = cbt_btree:open(ReaderRoot, Fd1).
{ok,{btree,<0.44.0>,
           {0,[],32},
           undefined,undefined,undefined,nil,snappy,1279}}
12> cbt_btree:lookup(SnapshotBtree, [a]).
[{ok,{a,1}}]

You can check that the database value is not change until we store the header:

13> {ok, Btree4} = cbt_btree:add(Btree2, [{a, 1}, {b, 2}]).
{ok,{btree,<0.35.0>,
           {4160,[],39},
           undefined,undefined,undefined,nil,snappy,1279}}
14> cbt_btree:lookup(Btree4, [a, b]).
[{ok,{a,1}},{ok,{b,2}}]
15> Root2 = cbt_btree:get_state(Btree4).
{4160,[],39}
16> Header2 = {1, Root2}.
{1,{4160,[],39}}
17> cbt_file:write_header(Fd, Header2).
ok
18> cbt_btree:lookup(SnapshotBtree, [a, b]).
[{ok,{a,1}},not_found]

ETS backend

Find here a simple usage of the ETS backend of cbt allowing you to store one database in an ETS.

1> cbt_ets:new(test).
test
2> {ok, Bt} = cbt_ets:open_btree(test, test).
{ok,{btree,test,cbt_ets,nil,identity,identity,
           #Fun<cbt_btree.1.30772535>,nil,none,1279,2558}}
3> {ok, Bt2} = cbt_btree:add(Bt, [{a, 1}]).
{ok,{btree,test,cbt_ets,
           {1,[],28},
           identity,identity,#Fun<cbt_btree.1.30772535>,nil,none,1279,
           2558}}
4>  cbt_ets:update_btree(test, test, Bt2).
true
5> {ok, SnapshotBtree} = cbt_ets:open_btree(test, test).
{ok,{btree,test,cbt_ets,
           {1,[],28},
           identity,identity,#Fun<cbt_btree.1.30772535>,nil,none,1279,
           2558}}
6> cbt_btree:lookup(SnapshotBtree, [a]).
[{ok,{a,1}}]
7> {ok, Bt3} = cbt_btree:add(Bt2, [{b, 2}]).
{ok,{btree,test,cbt_ets,
           {2,[],36},
           identity,identity,#Fun<cbt_btree.1.30772535>,nil,none,1279,
           2558}}
8> cbt_ets:update_btree(test, test, Bt3).
true
9> cbt_btree:lookup(SnapshotBtree, [a, b]).
[{ok,{a,1}},not_found]
10> {ok, SnapshotBtree2} = cbt_ets:open_btree(test, test).
{ok,{btree,test,cbt_ets,
           {2,[],36},
           identity,identity,#Fun<cbt_btree.1.30772535>,nil,none,1279,
           2558}}
11> cbt_btree:lookup(SnapshotBtree2, [a, b]).
[{ok,{a,1}},{ok,{b,2}}]i

Custom storage backend

CBT provides you 2 different backends by default:

cbt_file: Backend to store data in a file
cbt_ets: Backend to store data in ETS.

But can use a custom backends to store the btree data if you need it. For example if you want to store the btree in a custom file backend when you want to change the data types or when you want to store the BTREE over a Key/Value store.

To do it just pass the backend module to the btree and give it the Reference (atom or pid) that have been created when initializing the backend. Have a look in the `cbt_ets' module for more informations.

benoitc / cbt