Webhook for Grafana Loki

[Rajmohan R]

• Presently Loki is not a supported receiver to push the events. I am trying to make use of Webhook with a layout confirming to the Loki json format as below

{
  "streams": [
    {
      "stream": {
        "label": "value"
      },
      "values": [
          [ "<unix epoch in nanoseconds>", "<log line>" ],
          [ "<unix epoch in nanoseconds>", "<log line>" ]
      ]
    }
  ]
}

Loki requires the timestamp field to be in Unix epoch in nanoseconds. But the events exported by this exporter have timestamps in RFC3339 combined date-time format.

• Is it possible to control the timestamp format through the configmap? Until Grafana Loki is supported directly as a receiver, this feature to specify the timestamp format through configmap will help.

[Rajmohan R]

Added another function at the end of https://github.com/opsgenie/kubernetes-event-exporter/blob/master/pkg/kube/event.go

func (e *EnhancedEvent) GetTimestampUnixNano() int64 {
	return e.FirstTimestamp.UnixNano()
}

Called this function in the layout section of the webhook receiver for Grafana Loki like this:

    receivers:
      - name: "loki"
        webhook:
          endpoint: "http://192.168.0.30:3100/loki/api/v1/push"
          headers:
            Content-Type: application/json
            User-Agent: "kube-event-exporter"
          layout:
            streams:
            - stream:
                source: event-exporter
                app: kube-api
              values:
              - - "{{ .GetTimestampUnixNano }}"
                - "Namespace: {{ .InvolvedObject.Namespace }}, {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}, Reason: {{ .Reason }}, Message: {{ .Message }} "

[Alexander "Ananace" Olofsson]

You can sort of cheat this as it is right now by converting the millisecond timestamp to nanoseconds without needing to modify the code. The loss of precision may very well be acceptable for most people.

Something like this works in my use-case;

receivers:
  - name: "loki"
    webhook:
      endpoint: 'http://loki:3100/loki/api/v1/push'
      headers:
        Content-Type: application/json
        User-Agent: kube-event-exporter
      layout:
        streams:
          - stream:
              kind: kube-event
              namespace: '{{ .InvolvedObject.Namespace }}'
              type: '{{ .Type }}'
            values:
              - - "{{ mul .GetTimestampMs 1000000 }}"
                - 'level={{ lower .Type | replace "normal" "notice" }} namespace={{ .InvolvedObject.Namespace }} object={{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }} reason={{ .Reason }} message={{ quote .Message }}'