Error when to use encryptionSecret and prometheus alertManager Template

[Yoshiki Fujikane]

What happened:
When we tried to deploy kube-prometheus-stack by using PipeCD, plan preview failed like this↓

What you expected to happen:

We want to succeed to do a plan preview.

How to reproduce it:

• Register k8s app with a helm chart kube-prometheus-stack
• use both secret management (from PipeCD) and custom alert template in values.yaml like below↓

values.yaml

alertmanager:
...
  config:
    receivers:
    - name: 'pagerduty-notifications'
      pagerduty_configs:
      - service_key: {{.encryptedSecrets.key}}
  templateFiles:
     template_2.tmpl: |-
       {{ define "slack.custom.v2.text" }}
       {{ range .Alerts }}
         {{ if eq .Status "firing" }}{{ .Annotations.description }}{{else}}{{ .Annotations.resolved_description }}{{end}}
       {{ end }}
       {{ end }}

       {{ define "slack.custom.v2.title" }} [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }} {{ .GroupLabels.severity }}] {{ .GroupLabels.alertname }} {{ end }}

Environment:

• piped version:
• control-plane version:
• Others:

[Yoshiki Fujikane]

This happens because the custom alert template is parsed as Golang's text/template when parsing the encryptionSecret (from PipeCD).

[Yoshiki Fujikane]

Workaround:

1. Use a Secret resource for .encryptedSecrets.key and don't use it on a file.
2. Surround the part of the alert templating string by {{``}}. The surrounded part is parsed as just a string in Golang's text/template
   ref: helm/helm#2798 (comment)