emqx / replayq

Generic on-disk persistent queue implementation for Erlang

Home Page:https://www.emqx.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ReplayQ

A Disk Queue for Log Replay in Erlang

Features

  • Queue items are written to segment files on disk to survive restart.
  • Batch popping items out of queue with size/count limit.
  • An ack/2 API is provided to record the reader position within a segment.
  • Add config option max_total_bytes to limit the total size of replayq logs

Usage Example

Mem Only

Q0 = replayq:open(#{mem_only => true}),
Q1 = replayq:append(Q0, [Binary1, Binary2]),
{Q2, AckRef, [Binary1]} = replayq:pop(Q1, #{bytes_limt => 1}),
ok = replayq:ack(Q2, AckRef).

Binary Queue Items

Q0 = replayq:open(#{dir => "/tmp/replayq-test", seg_bytes => 10000000}),
Q1 = replayq:append(Q0, [Binary1, Binary2]),
{Q2, AckRef, [Binary1]} = replayq:pop(Q1, #{count_limit => 1}),
ok = replayq:ack(Q2, AckRef).

User Defined Queue Items

Q0 = replayq:open(#{dir => "/tmp/replayq-test",
                    seg_bytes => 10000000,
                    sizer => fun({K, V}) -> size(K) + size(V) end,
                    marshaller => fun({K, V}) -> term_to_binary({K, V});
                                     (Bin)    -> binary_to_term(Bin)
                                  end
                   }),
Q1 = replayq:append(Q0, [{<<"k1">>, <<"v1">>}, {<<"k2">>, <<"v2">>}]),
{Q2, AckRef, [{<<"k1">>, <<"v1">>}]} = replayq:pop(Q1, #{count_limit => 1}),
ok = replayq:ack(Q2, AckRef).

Offload mode

In offload mode, the disk queue is only used to offload queue tail segments. Add offload => true to Config for replayq:open/1.

Volatile mode

Using offload => {true, volatile} in the Config for replayq:open/1 will make it unconditionally clear any previous segments and commit file in the given work directory. Also, it will not dump in-memory data back to the disk when closing.

About

Generic on-disk persistent queue implementation for Erlang

https://www.emqx.io

License:Apache License 2.0


Languages

Language:Erlang 98.9%Language:Makefile 1.1%