Rate limit ineffective for kustomization reconciliation events with time stamps

[Luke Schulz]

Issue

Duplicate alerts sent to external service for successful Kustomization reconciliation with an unchanged revision.

Consequences

In our case, the github Provider would start to fail due to GitHub API rate limiting being applied.

To reproduce

enable rate limiting in notification controller

configure a Kustomization.interval to be less than the notification controller’s --rate-limit-interval

configure Alert for the Kustomization

Reason

The fluxcd/notification-controller provides rate limiting of duplicate events. The notification-controller uses Event.message as one of the fields to determine uniqueness. On successful reconciliation, the fluxcd/kustomize-controller emits an Event object with Event.message containing nanosecond precision duration of how long the reconcile took. The message has too much variance for the rate limiter implementation to meaningful dedupe Alerts.

Here are two example messages for the same Kustomization revision that successfully reconciled:

Reconciliation finished in 2.084434647s, next run in 1m0s

and

Reconciliation finished in 2.075916724s, next run in 1m0s

Version info

Flux version: 0.41.2
helm-controller: v0.31.2
image-automation-controller: v0.31.0
image-reflector-controller: v0.26.1
kustomize-controller: v0.35.1
notification-controller: v0.33.0
source-controller: v0.36.1