Simple Cloud Function that sets desired GCE Instance and Disk labels based on name match. Born as workaround to GKE limitation where it does not support setting GCE labels on the NodePools. Meaning that billing has a cluster level granularity, but not have a Node Pool level granularity.
Implemented in Go + Cloud Functions + PubSub + Cloud Scheduler
Cost management in large projects running GKE can be a challenge. While, there is a Metered Usage feature, it is too far removed from the rest of the billing. Instead, in plenty of the cases, it is possible to allocate cost to a team/product/division based on the instance name which is derived from a node pool. Unfortunately, GKE provides no such feature to label GCE instances serving as GKE nodes with labels. This Cloud Function solves this short-comming in a brute force manner, by re-evaluating labels on all the instances in the selected projects and adjusting them to be in line with the desired labels.
Specify projects, labels to match and labels to set in configuration.yaml
Example:
projects:
- gcp-project-1
- gcp-project-2
labels:
search:
team: search
purpose: hosts-search-applications
system:
team: devops
purpose: logging-and-monitoring
transcoding:
team: digital
purpose: transcode-videos
preemptible: "true"If you have an instance with the name gke-prod-transcode-videos-7ebf06e4-jds5, it will match transcode and set relevant labels.
Map map[team:digital purpose:transcode-videos preemptible:true] is not a subset map[goog-gke-node:]
Cloud Scheduler trigger PubSub Event every minute -> GKE Instance Auto Labeler Consumers PubSub Event > Cloud Function reconciles labels
# Prepare environment
go mod init github.com/dmitri-lerko/gke-instance-auto-labeler
go mod tidy
go build
# Create topic
gcloud pubsub topics create --labels=team=devops \
reconcile-labels-every-minute --project=<INSERT PROJECT NAME>
# Create service account
gcloud iam service-accounts create gke-instance-labels
# Get Organization ID
gcloud organizations list
# Create Organization level Custom Role with the least-privilege
gcloud iam roles create GKEInstanceLabelSetter --organization=<organization-id> \
--file=custom-role.yaml
# Bind Custom Role to the Service Account
gcloud organizations add-iam-policy-binding <organization-id> \
--member=serviceAccount:gke-instance-labels@<INSERT PROJECT NAME>.iam.gserviceaccount.com \
--role organizations/<organization-id>/roles/GKEInstanceLabelSetter
# Manually deploy
gcloud beta functions deploy GKEInstanceLabels --region <INSERT REGION> \
--runtime go111 --trigger-topic=reconcile-labels-every-minute \
--allow-unauthenticated --memory=128 --update-labels="team=devops" \
--service-account=gke-instance-labels@<INSERT PROJECT NAME>.iam.gserviceaccount.com \
--project=<INSERT PROJECT NAME> --timeout=540s
# Add hourly schedule
gcloud scheduler jobs create pubsub reconcile-labels-every-minute \
--schedule "* * * * *" --topic reconcile-labels-every-minute \
--message-body "Reconcile GKE Instance Labels every minute" \
--project <INSERT PROJECT NAME>- Function adds new labels where missing and overrides where present, but different
- Function also updates labels on the disk attached to the instance (disk must have the same name as the instance)
- Function does nothing when existing labels match
- GKE has it's own Label reconcillation loop, therefore it is advised to not try and control labels set on the cluster level with this tool, you will see frequent label updates
- The code is not defensive, but instead, it simply returns error where it can't handle edge cases
- This function will match non-GKE instances too as it currently does not filter by GKE labels
This function needs following GCP permissions to operate:
- compute.disks.get
- compute.disks.setLabels
- compute.instances.list
- compute.instances.setLabels
- compute.zones.list
You can run the application locally by invoking:
go run main/main.go