We need a data sink node service that runs on a small server device || phone
truedat101 opened this issue · comments
Problem
- We want a place to store either small mounts of data or large amounts of data (depending on the frequency of data sync and the kind of data)
- We want the first choice to be a local data sink node
- We want the data sink nodes to advertise themselves locally
- We want the data sink nodes to use security for authenticating clients (acl list, PKI, or other mechanism using secure transport)
- We want the data sink transactions to be reliable if required, however, default will be best effort reliability
- We want the data sink nodes to have the ability to sync to upstream data services as an option (i.e. s3, onedrive, dropbox, etc...)
- We want a data sink node to have the ability to visualize its data (#7 )
Definitions
- a data sink node: any destination that can store data on behalf of the user/device(s)
- data sink storage type: either blob or time series entry. Time series entry can be a metric or a log entry. A blob will be a data blob of some size, typically for larger text docs or binary such as an image.
- data sink capacity: the published capacity available for storage, typically will be finite, and style of storage may be a ring buffer if storage is extremely limited (i.e. might only store the most recent 100K of data)
- data sink api: the api exposed to applications to allow storage and retrieval of data
- data sink auth: the authentication method presented to applications to store/retrieve data
- data sink client: any device or app that sends data to sink
Open Questions
- How to sink smartly so that already previously sync'd data isn't sent
- Do we leave reliability of data sink to the client?
Use fluentd apis and the GELF format (https://graylog.org/post/log-formats-a-complete-guide/) used in graylog. Gelf is super flexible for a data unit.
We should focus on something that runs on linux first, and then worry about developing a phone based service.
I am adding my forked repo of expresscpp to make it possible for us to bootstrap a C++ project easily.
This needs some fixes, as there were upstream bugs. Will pull those in.