Abel client emits metrics to a kafka (0.10) topic. Each metric has a key and a value. Every key has a name, tags and timestamp. A value is of aggregate type. Aggregate is anything which (sort of commutative semigroup) can be folded with another aggregate of same type.
What abel consumer does is very simple, it regularly reads from kafka queue and merges all those metrics with same (exactly same) key and stores it to underlying rocksdb.
Add the library to build.sbt (published to sonatype)
"com.indix" %% "abel-core" % "1.1.8"
And the client emission of metrics looks like,
val client = AbelClient(AbelConfig("<host>", "<topic>"))
client.send(Metric(Key(name = "unique-upcs"), UniqueCount(product.getUpc)))
Abel currently supports
Count - Exact count of occurences
UniqueCount - Approximate counts using HLL+
Average - Exact average
Pair - Take 2 aggregates and club them together
By - Take enumerable number of key and aggregate of same type (Very useful to capture aggregate for various statuses which are queried together)
To query
http://<endpoint>/metrics?name=<name>&tags=<tag1>&tags=<tag2>&duration=<duration>&startTime=<startTime>&endTime=<endTime>
If the metric is emitted with tags, it has to be queried with tags (Work in progress to add explosion rules to also aggregate without tags)
- Support for percentiles with QDigest.
- Auto rollups for coarser timestamps.
- Integrate with distributed KV store.