yocto-db

A data-stream management system.

Build & Run

rebar clean compile
erl -pz ebin deps/*/ebin

Test

rebar eunit

Documentation

rebar doc

Supervisor Hierarchy

+ydb_sup
|
+---+ydb_input_stream_sup
    |
    +---+ydb_input_stream
    |   |
    |   +---+ydb_{file,socket}_input
    |       |
    |       +ydb_branch_node
    |
    +ydb_query_sup
    |
    +---+ydb_query

The ydb_input_stream_sup module supports registering input streams either from a file or a socket. The input stream process also spawns a branch node process and adds it as a subscriber.

IO Definitions

File Input

{ydb_file_input,
    child_node
  , filename :: string()
  , batch_size :: integer()
  , poke_freq :: integer()
}

File Output

{ydb_file_output, child_node, filename :: string()}

Socket Input

{ydb_socket_input,
    child_node
  , port_no :: integer()
  , socket :: port()
  , acceptor :: pid()
}

Socket Output

{ydb_socket_output,
    child_node
  , port_no :: integer()
  , socket :: port()
  , address :: term()
}

Query Definitions

Register a timer that pokes the intermediary plan nodes at each tick, and store the reference in the supervisor state. This keeps the plan nodes up to date on the current time so they how to store partial results.

Select

{ydb_select, child_node, predicate :: ydb_clause()}

Project

The column specification is either just the column name atom() or a tuple containing the current column name and the new name for the column {atom(), atom()}.

{ydb_project, child_node, columns :: [atom() | {atom(), atom()}]}

Joins

{ydb_join_node, left_node, right_node, predicate :: ydb_clause()}

Set Union

{ydb_union_node, left_node, right_node}

Set Difference

{ydb_diff_node, left_node, right_node}

Min

{ydb_min, child_node, column :: atom() | {atom(), atom()}}

Max

{ydb_max, child_node, column :: atom() | {atom(), atom()}}

Sum

{ydb_sum, child_node, column :: atom() | {atom(), atom()}}

Count

{ydb_count, child_node, column :: atom() | {atom(), atom()}}

Average

{ydb_avg, child_node, column :: atom() | {atom(), atom()}}

Variance

{ydb_var, child_node, column :: atom() | {atom(), atom()}}

Standard Deviation

[ydb_stddev, child_node, column :: atom() | {atom(), atom()}}

Predicate Format

Boolean Operators

{ydb_and, clauses :: [ydb_clause()]}
{ydb_or, clauses :: [ydb_clause()]}
{ydb_not, clause :: ydb_clause()}

ydb_clause() :: #ydb_cv{} | #ydb_cc{} | #ydb_and{} | #ydb_or{} | #ydb_not{}

Comparison Operators

Comparing a column to a value.

{ydb_cv,
    column :: atom()
  , operator :: compare()
  , value :: term()
}

Comparing two columns together.

{ydb_cc,
    left_col :: atom()
  , operator :: compare()
  , right_col :: atom()
}

Comparison operators.

compare() :: 'gt'  | 'lt'  | 'eq'
           | 'lte' | 'gte' | 'ne'

Table Formats

Rows in ETS tables are represented as tuples. All the rows in a given ETS table should be in the same format, which will be one of the following:

Relation Table

ydb_rel_tuple() :: {
    {'row_num', RowNum :: non_neg_integer()}
  , Tuple :: ydb_tuple()
}.

RowNum serves as a unique id for each row.

Synopsis Table

ydb_syn_tuple() :: {
    {Op :: atom(), Timestamp :: non_neg_integer()}
  , Tuple :: ydb_tuple()
}.

Op is the name of the aggregate (e.g. 'sum' or 'count').

Diff Table

ydb_diff_tuple() :: {
    {Diff :: diff(), Op :: atom(), Timestamp :: non_neg_integer()}
  , Tuple :: ydb_tuple()
}.

Op is the name of the aggregate, as above (e.g. 'sum' or count'). Diff indicates whether the tuple is to be inserted or deleted.

diff() :: '+' | '-'.

visemet / yocto-db