A demo project showing how to use Prometheus to monitor a machine learning system.
You should be able to view the dev
and prod
distributions on Grafana in real time.
Run python train.py
and python deploy.py
in seperate terminals and leave them running.
Follow the Starting Prometheus guide, using the prometheus.yml
in this repository as the config file when starting Prometheus locally.
Follow the Install Grafana guide to both install and start Grafana locally.
You can then add a Prometheus data source and create a panel. Create two panels with the following queries:
PromQL query for dev
distribution
model_feature_bucket{env="dev"}
PromQL query for prod
distribution
model_feature_bucket{env="prod"}
Select Heatmap
for both Visualization
and Format
, legend
to {{le}}
and Data Format
to Time series buckets
, like so:
Grafana doesn't allow for alerts to be set on heatmaps.
One option for detecting data drift is to write an exporter, which takes the exported metrics (on ports 8000
and 8001
in the demo), augmenting them (with e.g. distance between dev
and prod
using the KS test) and exposing them on another endpoint.
For more information on exporters, see Prometheus - Writing Exporters.