kelindar / column

High-performance, columnar, in-memory store with bitmap indexing in Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to stream filtered values ?

kant777 opened this issue · comments

Imagine I get a stream of messages like below from a incoming socket

{"name": "Alice", "grandslams": 20, "country": "switzerland"}
{"name": "Bob", "grandslams": 10, "country": "spain"}
{"name": "Charlie", "grandslams": 12, "country": "serbia"}

Now I want to filter all the messages that satisfies a certain criteria and push it to an outgoing socket in a streaming fashion. Like say stream all messages where grandslams > 10 and country = "spain". While I am aware few ways to implement this I wonder how to do it effectively using this library?

This library might not be the best tool for the job. Now, if you still want to implement something like this, I would do it in a micro-batch fashion:

  1. When receiving messages, keep inserting them into the database
  2. Periodically (every N milliseconds), perform the query, flush the messages and potentially delete the data.