Add rule_files configuration for the prometheus CRD to make prometheus watch other directories except those are created by the prometheus operator.

[JAESANGPARK]

Component(s)

Prometheus, PrometheusRule

What is missing? Please describe.

If the total amount of prometheusrules are over the limitation of k8s configmap (1Mib), then the operator create one more configmap and add additional volumes, volumeMounts configuration for the prometheus pod.

This leads the prometheus pod to be restarted. (changes under .spec leads for the pod to be restarted. )
As you know, as prometheus pod restarts the alerts are reset.
So now if I add some of prometheus rules, then it could lead all the alerts those are firing get reset.

My first work-around is mounting additional volumes where some of rulefiles those prometheus could comprehend are stored for the prometheus pod. (like /etc/prometheus/rules/custom-additional-rules)
But, the prometheus would watch only directory that the prometheus operator creates. (/etc/prometheus/rules/-rulefiles-)
The configuration of the prometheus is like below

global:  
  evaluation_interval: 15s  
  ...
rule_files:  
- /etc/prometheus/rules/<prometheus pod>-rulefiles-0/*.yaml  

According to above configuration, I have to mount rulefiles on the /etc/prometheus/rules/-rulefiles-0, but that is not allowed in the k8s volume (The mountPath should be unique).

My 2nd work-around is copying the rulefiles to the /etc/prometheus/rules/-rulefiles-0
The file system "/etc/prometheus/rules/-rulefiles-0" is mounted as read-only.
So I can't copying anything to there.

Describe alternatives you've considered.

My suggestion: How about adding the rule_files configuration to set custom rulefiles directory?

prometheus.yaml

...
rule_files:
- /etc/prometheus/rules/db/*.yaml
  /etc/prometheus/rules/frontend/*.yaml
  /etc/prometheus/rules/custom/*.yaml

then the prometheus.env.yaml would be renderd like below

global:  
  evaluation_interval: 15s  
  ...
rule_files:  
- /etc/prometheus/rules/prometheus-app-prometheus-rulefiles-0/*.yaml     # default
  /etc/prometheus/rules/db/*.yaml  #custom
  /etc/prometheus/rules/frontend/*.yaml   #custom
  /etc/prometheus/rules/custom/*.yaml   #custom

Environment Information.

Environment

Kubernetes Version: v1.20.7
Prometheus-Operator Version: v0.53.1