use case: visualise metrics stored as json files in S3
AmitKumarDas opened this issue · comments
ProblemStatement: As an automation engineer I collect various metrics as json format & upload them to S3. I want to visualise the same on Grafana like visualisation tools running in some K8s setup.
High Level Design Drafts - 0
This design is split across multiple custom resources:
- [1] Fetch metrics in json files from S3
- Command resource should delete resources that were created by it earlier
- Command custom resource to fetch files from S3
- This Command resource should run every day once (configurable)
- A TTL based ConfigMap can act as a lock to reconcile Command after 24 hours only
- Command resource should wrap each metrics file as a ConfigMap with labels
- Any failure in reconciling Command resource should be retried in some interval
- Command status should say when the last successful reconciliation was done
- Command resource will apply a new custom resource called
PrometheusLoader
- [2] Upload json files containing metrics to Prometheus
- Implement
PrometheusLoader
as a custom resource - It should translate all the relevant ConfigMaps as Prometheus metrics & upload the later
- Schema to translate data from json structure to prometheus metrics needs to be designed
- This schema itself will be a json structure that can be wrapped inside a ConfigMap
- Any failure to reconcile PrometheusLoader will be retried
- [3] Implement a TTL based Lock
- Existence of a ConfigMap is considered as a lock
- This ConfigMap can be created by other controllers with exact labels
- This ConfigMap should also have a label with Time To Live i.e. TTL
- The corresponding controller will delete this ConfigMap once its TTL expires
NOTE:
- All the custom resources will be implemented as ConfigMaps
- This will be done to avoid the need of applying Custom Resource Definition(s)