custommetrics
This demo illustrates how to use a generic Cloud Function to trigger on already existing PubSub topic, extract data from its payload, and publish it to Google Stackdriver as a custom metric without altering your original data pipeline.
Context
If you have done any distributed development on GCP you've probably used PubSub to connect two or more data/event processing components (e.g. IoT Gateway to BigQuery or GCE-based batch process to Dataflow processing etc.). When I build these kind of integrations I often find myself wishing I could "take a pick" at one of the metrics in the data flowing through the system.
{
"source_id":"device-1",
"event_id":"63404713-76c8-412b-a8fc-49f35409a977",
"event_ts":"2019-05-14T05:48:13.132652Z",
"metric": {
"label":"friction",
"value":0.2789,
},
}
Assuming for example that your data published to PubSub topic has the above shape, in this demo I will illustrate how you can easily monitor the deviation of friction
per each device
over time without the need to manage additional infrastructure.
Additionally, I will show you how you can create monitoring policy to alert you when the monitored metric falls outside of the pre-configured range.
Note, this will only work on JSON PubSub payloads
Configuration
Assuming the above JSON payload shape on your PubSub topic, there are few variables we need to define first is the name of the PubSub topic on which you want to trigger
TOPIC="iotevents"
We also need to define the Stackdriver metric type (custom.googleapis.com/metric/*
) where the last part is the type of metric we will be tracking (e.g. custom.googleapis.com/metric/friction
for friction) and the payload selector paths for SRC_ID_PATH
, VALUE_PATH
, TIME_PATH
.
FVARS="METRIC_TYPE=custom.googleapis.com/metric/friction"
FVARS+=",SRC_ID_PATH=source_id"
FVARS+=",VALUE_PATH=cpu_used"
FVARS+=",TIME_PATH=event_ts"
Few things to keep in mind:
- The value of
SRC_ID_PATH
selector must uniquely identity of the source of this event across all events - The value of
VALUE_PATH
select must be numeric (int or float) TIME_PATH
is optional. If not set it will be set to event processing time. But if set, it must be in RFC3339 format
Deployment
Once you have these variables defined, you can deploy the Cloud Function
gcloud functions deploy custommetrics \
--entry-point EventProcessor \
--set-env-vars=$FVARS \
--memory 256MB \
--region us-central1 \
--runtime go111 \
--trigger-topic $TOPIC \
--timeout 540s
If everything goes well you will see a confirmation
$: bin/deploy
Deploying function (may take a while - up to 2 minutes)...done.
...
status: ACTIVE
versionId: '3'
Monitoring
In Stackdriver now you can use Metric Explorer to build a time-series chart of the published data. First, use the resource type and metric finder to paste your metric type. Using the above deployment as an example that would be custom.googleapis.com/metric/friction
.
Now you can use the Group By
to display the time series per source_id
and optionally specify the Aggregator in case you want to display mean
for more volatile metrics.
More details on Stackdriver metrics selection here
Also, make sure to consider the Stackdriver Monitoring limits on custom metrics before implementing this. Specifically, the fact that the rate at which data can be written to a single time series is 1 point per min.
Alerts
Once you have custom metric monitored in stack driver you can easily create an alert policy. Many options there with regards to the duration, single or multiple sources, and percentages vs single values.
Once configured though, when the monitored metrics violates that policy you can get notifications about an incident
Obviously this approach is meant to replace a proper monitoring solution. Still, I hope you found it helpful.
Disclaimer
This is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.