Backups to azblob with AZBLOB_MANAGED_IDENTITY

[shivtej-opsverse]

I have a clickhouse pod running with the clickhouse-backup container running as a sidecar.
Following are the environment variables i have used for the backup-container

                    - name: API_CREATE_INTEGRATION_TABLES
                      value: "true"
                    - name: BACKUPS_TO_KEEP_REMOTE
                      value: "5"
                    - name: REMOTE_STORAGE
                      value: azblob
                    - name: CLICKHOUSE_HOST
                      value: 127.0.0.1
                    - name: AZBLOB_ACCOUNT_NAME
                      value: <storage-account-name>
                    - name: AZBLOB_USE_MANAGED_IDENTITY
                      value: "true"
                    - name: AZBLOB_CONTAINER
                      value: <container-name>

Both the k8s cluster of clickhouse pod and the storage account belong to the same resource group.

After this I have created a Managed Identity and gave it access as Storage Blob Data Contributor to the the above resource group.

When i try to upload a backup it says

2024/03/07 10:50:02.790773 error API /backup/actions error: can't connect to azblob: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/runner/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.15.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationFailure) =====
Description=This request is not authorized to perform this operation.
RequestId:6787088e-a01e-0005-207d-70b825000000
Time:2024-03-07T10:50:02.8113470Z, Details: 
   Code: AuthorizationFailure
   PUT https://<container-name>.blob.core.windows.net/<storage-account-name>?restype=container&timeout=901
   Authorization: REDACTED
   User-Agent: [Azure-Storage/0.15 (go1.20.5; linux)]
   X-Ms-Client-Request-Id: [f24b3725-1d32-4d48-50c9-28293882ad5f]
   X-Ms-Version: [2020-10-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [246]
   Content-Type: [application/xml]
   Date: [Thu, 07 Mar 2024 10:50:02 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [f24b3725-1d32-4d48-50c9-28293882ad5f]
   X-Ms-Error-Code: [AuthorizationFailure]
   X-Ms-Request-Id: [6787088e-a01e-0005-207d-70b825000000]
   X-Ms-Version: [2020-10-02]

Am I doing something wrong here?
How does the clickhouse backup understand which managed identity to use?
Any help would be appreciated.
Thanks

[Eugene Klimov]

managed identity was external contributed
#267
look implementation details in
https://github.com/Altinity/clickhouse-backup/blob/master/pkg/storage/azblob.go#L77-L107

@roman-vynar could you suggest us, how should works Azure Managed Identify?

[Roman Vynar]

Hello,
My working config

general:
  remote_storage:      azblob
...
azblob:
  endpoint_suffix:      core.windows.net
  account_name:         foobar
  account_key:          ""
  sas:                  ""
  use_managed_identity: true
  container:            backups
  path:                 "clickhouse/binary"
...

Also I assign Storage Blob Data Owner to a VM principal_id with a scope of container's resource_manager_id.
Terraform:

resource "azurerm_role_assignment" "dbnode_backup_access" {
  scope                = azurerm_storage_container.backups.resource_manager_id
  role_definition_name = "Storage Blob Data Owner"
  principal_id         = azurerm_linux_virtual_machine.dbnode1.identity[0].principal_id
}

[Eugene Klimov]

@shivtej-opsverse please notify if suggested solutions #866 (comment) works for you?

[shivtej-opsverse]

Hey @roman-vynar can you clarify a little more on how you are running clickhouse and the backup.
Are you running both of these in a VM and assigning it a role Storage Blob Data Owner with the scope of the above resource manager which actually manages the storage account foobar?

[Roman Vynar]

Running both in Docker containers: clickhouse runs in docker permanently and clickhouse-backup is kicked off via docker run.
The scope is for backups container in the storage account foobar, not the whole storage account scope.

[shivtej-opsverse]

Thanks @roman-vynar / @Slach i was able see which managed identity my pod was using by the command az aks pod-identity list and add that managed identity as a storage blob data owner role to the container and that worked.
Closing the issue.