How to stream filtered values ?
kant777 opened this issue · comments
kant777 commented
Imagine I get a stream of messages like below from a incoming socket
{"name": "Alice", "grandslams": 20, "country": "switzerland"}
{"name": "Bob", "grandslams": 10, "country": "spain"}
{"name": "Charlie", "grandslams": 12, "country": "serbia"}
Now I want to filter all the messages that satisfies a certain criteria and push it to an outgoing socket in a streaming fashion. Like say stream all messages where grandslams > 10 and country = "spain"
. While I am aware few ways to implement this I wonder how to do it effectively using this library?
Roman Atachiants commented
This library might not be the best tool for the job. Now, if you still want to implement something like this, I would do it in a micro-batch fashion:
- When receiving messages, keep inserting them into the database
- Periodically (every N milliseconds), perform the query, flush the messages and potentially delete the data.