Teams integration via Proxy

[giladk1221]

Hi,
We are trying to configure teams integration but the messages don't appear in the teams channel.
We configured the notification controller to work with http_proxy because we are working In an offline environment.
Everything is configured well as mentioned in the doc(provider, secret with teams webhook URL, alert)

We don't see any relevant logs in the notification controller container.

Is it possible to enable debug mode to see more details as to why the connection is not working as expected?

[Max Jonas Werner]

You need to set .spec.proxy explicity in your Provider: https://fluxcd.io/flux/components/notification/providers/#https-proxy

edit: Some more context: n-c doesn't use net/http's DefaultTransport, therefore the http_proxy environment variable isn't consumed by most Providers.

[giladk1221]

Thank you for the response. After adding the .spec.proxy in the provider YAML, the issue persists. I cannot find any relevant logs in the Notification-Controller container; I believe it's not passing through the proxy and is getting blocked.

Is it possible to increase the debug mode in the container? I checked the Webhook URL for Teams in a different container with access to the curl command, and it's functioning without any issues. I can see new messages in the Teams channel.

@makkes
@yuval987

[Max Jonas Werner]

If you can't find any log messages like the following then it is unlikely that n-c didn't successfully send it.

$ $ k logs -n flux-system -l app=notification-controller | jq 'select(.level=="error")'                                                                                                                                                                                                   
{                                                                                                                                                                                                                                                                                                      
  "level": "error",                                                                                                                                                                                                                                                                                    
  "ts": "2023-11-24T14:44:58.857Z",                                                                                                                
  "logger": "event-server",                                                                                                                        
  "msg": "failed to send notification",                                                                                                            
  "eventInvolvedObject": {                                                                                                                         
    "kind": "Kustomization",                                                                                                                                                                                                                                                                           
    "namespace": "flux-system",                                                                                                                    
    "name": "flux-system",                                                                                                                         
    "uid": "dfe6846f-416c-427e-80a3-244f3840c4f5",                                                                                                 
    "apiVersion": "kustomize.toolkit.fluxcd.io/v1",                                                                                                
    "resourceVersion": "357617"                                                                                                                    
  },                                                                                                                                               
  "alert": {                                                                                                                                                                                                                                                                                           
    "name": "msteams-test",                                                                                                                        
    "namespace": "flux-system"                                                                                                                     
  },                                                                                                                                               
  "error": "postMessage failed: failed to execute request: context deadline exceeded"                      
}

[giladk1221]

Yes, we don't see anything in the logs.
It's very strange because we are following the documents with all the steps.
Is it possible to increase debug mode? To understand where it's tried to send the message and blocked?
Because the team's channel working successfully, I sent a message from a different container with a curl command from the same k8s cluster with the same proxy configuration.
@makkes

[Max Jonas Werner]

Is it possible to increase debug mode? To understand where it's tried to send the message and blocked?

If you see no message in the logs such as the one I pasted above but still don't get any message in your Teams channel then there's two potential reasons for that:

1. The message has been submitted successfully by notification-controller but something on the receiving end is messed up. In this case there really is nothing that n-c could do in terms of debugging.
2. The message has not been submitted by notification-controller because no Alert caught it.

To make sure that (2) isn't the case, check the logs of the notification-controller pod for a message like this:

{"level":"info","ts":"2023-12-12T16:54:00.438Z","logger":"event-server","msg":"Dispatching event: Reconciliation finished in 324.766878ms, next run in 5m0s","eventInvolvedObject":{"kind":"Kustomization","namespace":"foo","name":"bar","uid":"b65a4948-12a5-41fd-b3fd-777beda389b8","apiVersion":"kustomize.toolkit.fluxcd.io/v1","resourceVersion":"415263856"}}

To further rule out notification-controller as the source of failure here, create a Provider of a different type, e.g. generic webhook and set up a simple web server logging requests.