linuxfood / sqlite-setab

Stream Processing on top of SQLite and ZeroMQ

sqlite-setab

Stream Processing on top of SQLite and ZeroMQ

This is a proof of concept stream processing "engine" built on top of SQLite. It's by no means complete, but you can do all kinds of wacky things with it already. Such as stream-joins, graphs of processing, etc, all with simple SQL.

Example output:

This was generated by running ./buck-out/gen/setab/stream_maker 6000. From here, I could start a second instance of setab to read the output of the first and do further computation. It's easy to imagine implementing streaming map-reduce or some such using this.

$ ./buck-out/gen/setab/setab
Initialized setab module..
Create debug..
arg:0 value:'tag TEXT'
arg:1 value:'latency INTEGER'
arg:2 value:'listen_port=6000'
arg:3 value:'batch_size=10'
arg:4 value:'window_size_ms=60000'
key='listen_port', value=6000
key='batch_size', value=10
key='window_size_ms', value=60000
table schema: CREATE TABLE x(ts INTEGER, tag TEXT, latency INTEGER);
Going to bind to: tcp://*:6000
Created test_table1..
Create debug..
arg:0 value:'tag_group TEXT'
arg:1 value:'latency_sum INTEGER'
arg:2 value:'latency_count INTEGER'
arg:3 value:'next_hop_service='tcp://localhost:6001''
key='next_hop_service', value='tcp://localhost:6001'
table schema: CREATE TABLE x(ts INTEGER, tag_group TEXT, latency_sum INTEGER, latency_count INTEGER);
Going to connect to: tcp://localhost:6001
Created test_table2..
Compiled query: 
SELECT ts, sum(latency), count(latency), group_concat(tag) FROM test_table1;

Compiled query: 
INSERT INTO test_table2 (ts,tag_group,latency_sum,latency_count) VALUES(?,?,?,?);
Closing cursor
ts: 1457341732991 sum:134518294 cnt:10 value: 'mass-blaster,horsey,horsey,blamo mac-n-cheese,blamo mac-n-cheese,mass-blaster,horsey,blamo mac-n-cheese,mass-blaster,horsey'
Performing 'insert' into test_table2
col[0]:1457341732991
col[1]:mass-blaster,horsey,horsey,blamo mac-n-cheese,blamo mac-n-cheese,mass-blaster,horsey,blamo mac-n-cheese,mass-blaster,horsey
col[2]:134518294
col[3]:10
Query finished.
Closing cursor
ts: 1457341738643 sum:308460 cnt:10 value: 'mass-blaster,wiz-kid thelma the cool,horsey,horsey,mass-blaster,wiz-kid thelma the cool,wiz-kid thelma the cool,wiz-kid thelma the cool,horsey,wiz-kid thelma the cool'
Performing 'insert' into test_table2
col[0]:1457341738643
col[1]:mass-blaster,wiz-kid thelma the cool,horsey,horsey,mass-blaster,wiz-kid thelma the cool,wiz-kid thelma the cool,wiz-kid thelma the cool,horsey,wiz-kid thelma the cool
col[2]:308460
col[3]:10
Query finished.
Closing cursor
ts: 1457341745742 sum:3402479066 cnt:10 value: 'blamo mac-n-cheese,merble,horsey,horsey,wiz-kid thelma the cool,blamo mac-n-cheese,blamo mac-n-cheese,mass-blaster,wiz-kid thelma the cool,horsey'
Performing 'insert' into test_table2
col[0]:1457341745742
col[1]:blamo mac-n-cheese,merble,horsey,horsey,wiz-kid thelma the cool,blamo mac-n-cheese,blamo mac-n-cheese,mass-blaster,wiz-kid thelma the cool,horsey
col[2]:3402479066
col[3]:10
Query finished.
Closing cursor
ts: 1457341754305 sum:212941 cnt:10 value: 'wiz-kid thelma the cool,mass-blaster,mass-blaster,wiz-kid thelma the cool,mass-blaster,blamo mac-n-cheese,horsey,mass-blaster,wiz-kid thelma the cool,horsey'
Performing 'insert' into test_table2
col[0]:1457341754305
col[1]:wiz-kid thelma the cool,mass-blaster,mass-blaster,wiz-kid thelma the cool,mass-blaster,blamo mac-n-cheese,horsey,mass-blaster,wiz-kid thelma the cool,horsey
col[2]:212941
col[3]:10
Query finished.
Closing cursor
ts: 1457341762145 sum:1068675698 cnt:10 value: 'blamo mac-n-cheese,mass-blaster,mass-blaster,mass-blaster,wiz-kid thelma the cool,wiz-kid thelma the cool,merble,merble,wiz-kid thelma the cool,blamo mac-n-cheese'
Performing 'insert' into test_table2
col[0]:1457341762145
col[1]:blamo mac-n-cheese,mass-blaster,mass-blaster,mass-blaster,wiz-kid thelma the cool,wiz-kid thelma the cool,merble,merble,wiz-kid thelma the cool,blamo mac-n-cheese
col[2]:1068675698
col[3]:10
Query finished.

Building

Building is only supported on ubuntu 16.04 or above right now. Setab requires Facebook's folly library, the setup script will install that in /usr/local.

1. Get CMake
2. bash ubuntu_setup.sh
3. mkdir build && cd build && cmake ..
4. make
5. ./setab/setab test.db ../setab/test1.json

If you successfully pull that off... Drop me a line. The code is absolutely riddled with bugs right now.

Building with Docker

This works now! This is probably the easiest way to build and run this.

sudo docker build -t real_deal .
sudo docker run -it real_deal /bin/bash
# Then follow from step 3 above.

License

Everything in here is MIT licensed except SQLite, which of course retains license. I included it because this virtual table implementation makes some very modest use of very new features.