Epic: Bundled Observability

[Danny Kopping]

This is the holding issue for the set of improvements we want to make based on RFC: Bundled Observability.

Goal: produce a separate Helm chart which can be installed with one script to observe a Coder deployment. Each sub-chart should contain at least one dashboard, alert, and runbook to cover its own functional requirements (i.e. the Grafana installation should have a dashboard, alert, runbook to enable operators to observe it). The Coder deployment should be covered by several dashboards, alerts & runbooks, and we should collect all telemetry signals as applicable (metrics, logs, traces, profiles).

Initial Requirements:

• Deploy the LGTM stack in our internal environment to setup & configure (https://github.com/coder/dogfood/issues/43)
• Create a new Helm chart for bundled observability
  • coder/coder#12798
  • coder/coder#12799
    • Enable configureable notification routes (maybe offer Coder deployment as notification target, display alerts in UI?)
  • coder/coder#12800
    • Enable configureable notification routes (maybe offer Coder deployment as notification target, display alerts in UI?)
  • coder/coder#12994
  • coder/coder#12995
  • coder/coder#12996
  • coder/coder#12997
• Create observability resources
  • #2
  • #3
  • Pull in dashboard(s) for Postgres
  • #4
  • #5
    • Include optional but recommended note about installing this chart in the k8s docs
  • #6

Eventual Requirements:

• Add & configure profiling chart, add Grafana datasource
• Add & configure tracing chart, add Grafana datasource

[Danny Kopping]

Adhoc task list:

General

• handle unmanaged resources (i.e. resources created but subsequently not managed by helm)
  • make monitoring.prod deletes the existing manifests before generating new ones, meaning that if a helm chart value is changed to exclude a resource which was previously present, its manifest will now be missing and therefore Flux will remove it
• find a way to include common patches & other configs in local/prod kustomizations
• prevent secrets from charts being detected by {en,de}crypt_secrets.sh https://github.com/coder/dogfood/pull/56
• add githook to detect manifest differences locally https://github.com/coder/dogfood/pull/56
• add linter to detect manifest differences before PR merge is allowed https://github.com/coder/dogfood/pull/57
• add differ to show the impact of the new manifests on the existing cluster
• https://github.com/coder/dogfood/issues/53

Prometheus / Alertmanager

• ignore container ports discovered with kubernetes_sd_config which do not expose metrics (i.e. Loki's gRPC ports) (use extraScrapeConfigs)
• alert source addresses should use FQDN (prometheus-server.monitoring.svc.cluster.local)
• include runbook link via label
• test out slack receiver
• optimise labels and unify with loki labels

Grafana

• dashboards persistence https://github.com/coder/dogfood/pull/54

Loki

• optimise labels
• scrape logs from all pods
• add cluster label to loki metrics

[Ben Potter]

Can we add this as a step in our Kubernetes install docs as Recommended, but optional? https://coder.com/docs/v2/latest/install/kubernetes

[Danny Kopping]

Can we add this as a step in our Kubernetes install docs as Recommended, but optional? https://coder.com/docs/v2/latest/install/kubernetes

Absolutely, will add an item