3rd-party controller notifications

[Daniel França]

I'm trying to setup notifications from tf-controller, so I followed what is mentioned here

I'm not sure if this falls here, or better on tf-controller... but my guess is that I'm just misconfiguring something

I have checked the resources and they seem to contain the correct patches:

kubectl -n flux-system describe crd alerts.notification.toolkit.fluxcd.io
...
                    Kind:
                      Description:  Kind of the referent
                      Enum:
                        Bucket
                        GitRepository
                        Kustomization
                        HelmRelease
                        HelmChart
                        HelmRepository
                        ImageRepository
                        ImagePolicy
                        ImageUpdateAutomation
                        OCIRepository
                        Terraform
...

kubectl -n flux-system describe crd receivers.notification.toolkit.fluxcd.io
...
                    Kind:
                      Description:  Kind of the referent
                      Enum:
                        Bucket
                        GitRepository
                        Kustomization
                        HelmRelease
                        HelmChart
                        HelmRepository
                        ImageRepository
                        ImagePolicy
                        ImageUpdateAutomation
                        OCIRepository
                        Terraform
                      Type:  string
...

kubectl -n flux-system describe clusterrole crd-controller-flux-system
Name:                               crd-controller-flux-system
Labels:                             app.kubernetes.io/instance=flux-system
                                    app.kubernetes.io/part-of=flux
                                    app.kubernetes.io/version=v0.34.0
                                    kustomize.toolkit.fluxcd.io/name=flux-system
                                    kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:                        PolicyRule:
  Resources                         Non-Resource URLs  Resource Names  Verbs
  ---------                         -----------------  --------------  -----
...
  *.infra.contrib.fluxcd.io         []                 []              [*]
...

My Alert specification is the following

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: general
  secretRef:
    name: slack-url
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: flux-cd
  namespace: flux-system
spec:
  summary: "Flux CD"
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: Kustomization
      name: '*'
    - kind: Terraform
      name: '*'

It works fine for GitRepositiory resources, but for Terraform resource I get no notification at all, and I don't see anything in the notification-controller logs, not even a mention whether a rule was not satisfied

Is there anything else I'm missing on the setup?

[Chanwit Kaewkasi]

It seems to be the correct configuration to me. I did try the similar setup and it worked.
Maybe you would try specifying apiVersion for your event source?

Below is my working config:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: terraform-slack
  namespace: default
spec:
  providerRef:
    name: slack
  eventSeverity: info
  eventSources:
    - apiVersion: infra.contrib.fluxcd.io/v1alpha1
      kind: Terraform
      name: '*'

[Daniel França]

Hey @chanwit Thanks for answering very quickly

I have tried adding the apiVersion, still no success:

The way I'm testing it is changing the Terraform path to an invalid path
Then I see it on the tf-controller logs:

{"level":"info","ts":"2023-01-17T19:18:09.392Z","msg":"clean up dir: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"saas-github","namespace":"flux-system"},"namespace":"flux-system","name":"saas-github","reconcileID":"d0f5928b-2c40-4581-8553-345180873591"}
{"level":"error","ts":"2023-01-17T19:18:09.415Z","msg":"Reconciliation failed after 15.440494373s, next try in 2h0m0s","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"saas-github","namespace":"flux-system"},"namespace":"flux-system","name":"saas-github","reconcileID":"d0f5928b-2c40-4581-8553-345180873591","revision":"master/bfca0740f9f0f164f9cc7ab86a29b7cad7cbe65d","error":"rpc error: code = Unknown desc = terraform path not found: stat /tmp/flux-system-saas-github/terraform/xsaas/github: no such file or directory"}

Which is what I expected, however, the notification-controller doesn't seem aware of this error at all
Is there a way that I can maybe debug who should be sending the data to the notification-controller?

[Chanwit Kaewkasi]

My next guess would be similar to your.

tf-controller needs the url of the notification controller and posts copies of events there.

[Daniel França]

Found the issue, I'm installing it manually using the manifest, not a helm chart
And it seems that the events-addr has only a default value when using the Helm Chart

So I had to manually add this to the list of arguments:

--events-addr=http://notification-controller.flux-system.svc.cluster.local.

[Chanwit Kaewkasi]

I'm glad it worked for you now. Shall we close this issue?

[Daniel França]

yes, but it would be nice to add it on the documentation

[Chanwit Kaewkasi]

ack. will address it.