Dagster 🪢 Azure Key Vault
This repo is an example of using Azure Key Vault as a Dagster resource.
I am open sourcing this both to get feedback and learn ... and to save you a lot of frustration (this took waaay too long for me to figure out)
If you have any questions, feedback, bugs, or improvements, create an issue - or contact me on the Dagster Slack, my name is Martin Carlsson
, or directly on martin@imus.dk
And don't forget to hit the ⭐️ button.
Setup repo on your local machine
Download repo
Start by forking this repo, hit the star icon ⭐️, and download the repo locally.
Install development environment
Use Remote development in Containers or install requirements directly pip install -r requirements.txt
pytest
Run Set environment variables:
export AZURE_KEY_VAULT_URI=
export AZURE_KEY_VAULT_SERVICE_PRINCIPAL_CLIENT_SECRET=
export AZURE_KEY_VAULT_SERVICE_PRINCIPAL_CLIENT_ID=
export AZURE_TENANT_ID=
Execute pytest:
Just run pytest
in the terminal.
Run Dagit
cd dagster_azure_key_vault
dagit
Config:
ops:
get_secret_from_azure_key_vault:
config:
azure_key_vault_secret_name: "ForTestingKeyVault"
resources:
key_management:
config:
azure_key_vault_service_principal_client_id:
azure_key_vault_service_principal_client_secret:
azure_key_vault_uri:
azure_tenant_id:
Setup and connect to Azure Key Vault
Create an Azure Key Vault
Create a key vault using the Azure portal
Create a secret
Open Key Vaults in Azure portal:
Select the Key Vault you just created:
Copy Vault URI:
You will need it when connecting to Azure Key Vault from Dagster.
Click on Secrets:
Click on +Generate/Import:
For testing, create the following secret:
- Name: ForTestingKeyVault
- Value: TestValidated
Create Service Principal
We will connect to Azure Key Vault via a Service Principal.
Register an application with Azure AD and create a service principal
Give the Service Principal access to Azure Key Vault:
Give Service Principal access to Azure Key Vault