Labels with special characters (".", "/") are not supported by Loki labelKeys

[Philipp Moeller]

Describe the bug

We use a ClusterLogForwarder to Loki with custom labels. The labels keys contain slashes and dots.

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: our-loki
      type: "loki"
      url: ...
      secret:
        name: ...
      loki:
        labelKeys:
        - kubernetes.labels.ourname.com/service-name
        - kubernetes.namespace_labels.ourname.com/center
  pipelines:
    - name: all-logs
      detectMultilineErrors: true
      inputRefs:
        - application
      outputRefs:
        - our-loki

Those labels do not end up in our loki instance. A look at the generated collector config explains why:

<label @CENTRAL_LOKI>
  <filter **>
    @type record_modifier
    <record>
      _kubernetes_namespace_labels_ourname_com/center ${record.dig("kubernetes","namespace_labels","ourname","com/center")}
      _kubernetes_namespace_name ${record.dig("kubernetes","namespace_name")}
    </record>
  </filter>

  <match **>
    @type loki
    @id central_loki
    line_format json
    ... 
    <label>
      kubernetes_labels_ourname_com/service-name _kubernetes_labels_ourname_com/service-name
      kubernetes_namespace_labels_ourname_com/center _kubernetes_namespace_labels_ourname_com/center
    </label>
    <buffer>
    ...
    </buffer>
  </match>
</label>

The resulting label is not valid in Loki (contains a slash) and the way the record is accessed is wrong as well. We could not get this to work using escapes.

Environment

• OpenShift 4.9
• Cluster Logging 5.5.5
• Loki 2.7

Expected behavior

Provide a way to specify labels with special characters as they are supported and used by k8s.

[Jeff Cantrill]

The resulting label is not valid in Loki (contains a slash)

My expectation is we solve this be replace the "slash" also with an underscore.

the way the record is accessed is wrong

Please clarify what this means? What are your expectations? What does it do that it should not?

[Jeff Cantrill]

@periklis do you know of other special chars we should consider? I assume the recommendation is to make the a "slash". I seem to recall that kubernetes only supports dots and slashes in a label anyway

[Philipp Moeller]

Please clarify what this means? What are your expectations? What does it do that it should not?

The generated record.dig instructions have too many levels, because the dot in the label name is treated as a separator.

This ${record.dig("kubernetes","namespace_labels","ourname","com/center") should be ${record.dig("kubernetes","namespace_labels","our name.com/center")

[OpenShift Bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[Philipp Moeller]

/remove-lifecycle stale

[Jeff Cantrill]

This should be resolved by #1931