azdevops - "could not list commit statuses: Request returned status: 401 Unauthorized"

[MarkStromme]

The notification controller fails to send notification because its unauthorised.

I have followed the documentation and created secret called azdevops (the PAT token has full access to everything):

apiVersion: v1
kind: Secret
metadata:
  name: azuredevops
data:
  token: <base64 encoded PAT token>

I have an alert and provider defined like this in "some-namespace" namespace:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: test
  namespace: some-namespace
spec:
  type: azuredevops
  address: https://dev.azure.com/<org>/<projectx>/_git/<repository>
  secretRef:
    name: azuredevops
---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: test
  namespace: some-namespace
spec:
  providerRef:
    name: test
  eventSeverity: info
  eventSources:
  - kind: Kustomization
    name: some-kustomization
    namespace: some-namespace

"some-kustomization" kustomization:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: some-kustomization
  namespace: some-namespace
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: some-repository
  path: ./apps/base/some-namespace
  prune: true

For some reason there were no updates on the commit statuses in Azure DevOps repo.
This is the log from notification controller when I trigger "some-kustomization" to reconcile:

{"level":"error","ts":"2022-05-13T06:45:17.591Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"Kustomization","name":"some-kustomization","namespace":"some-namespace","error":"could not list commit statuses: Request returned status: 401 Unauthorized"}

I have tried to make this work for couple of days now without any luck... Any thoughts?

[Stefan Prodan]

We do have e2e tests for Azure DevOps commit status updates that run on Azure infra, the last test run was successful https://github.com/fluxcd/flux2/actions/workflows/e2e-azure.yaml.

I suspect the token does not have the right permissions, maybe @somtochiama or @phillebaba could say more about which permissions are needed.

[MarkStromme]

@stefanprodan The PAT has full access, so I don't think that is the issue.

Note:
The provider and alert has the status "Initialized", no other message.

Edit:
Could you please provide documentation for Azure DevOps repos with an example? Would be very helpful.

Edit1:
Verified that PAT works and have access to list status by running:
curl -u :${PAT} https://dev.azure.com/<org>/<project>/_apis/git/repositories/<repository-id>/commits/<commit-id>/statuses\?api-version\=6.0

Edit2:
Added PAT in secret in this format: [e-mail]:PAT base64 encoded which now gives me different error:

{"level":"error","ts":"2022-05-13T12:55:25.858Z","logger":"event-server","msg":"failed to send notification","reconciler kind":"Kustomization","name":"some-kustomization","namespace":"some-namespace","error":"could not list commit statuses: TF400813: The user 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource."}

Edit3:
Tested authentication using Postman - got 401 when using Bearer in header, got correct response when using Basic in header with base64 encoded :$(PAT). What type of authentication does the notification controller use in header?
I see in microsoft documentation that they are using Basic authentication header: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#use-a-pat

[MarkStromme]

My bad, i encoded the token wrong.

Generating the token as follows solved the problem:
echo -n <PAT> | base64

The status in Azure DevOps repos now updates accordingly.