MLflow Webhooks End-to-End Template

Realtime Prediction Endpoints

Databricks MLflow Webhooks are a powerful new feature that allows customers to go to the next level in terms of MLOps best practices and CICD integration
This repo aims on showcasing a sample use case where we deploy a realtime predicition endpoint.
We consider a use case where a model was trained & tracked using Databricks & MLflow. With each model stage change, we want to trigger an Azure DevOps* build pipeline, which will download the model artifacts from MLflow, buld a Docker container to package them and push this container to Azure Container Registry.
It contains:
- Terraform IaC code for creating all the infrastructure needed: MLflow Webhooks, Azure Functions, Azure Kubernetes Service clusters, Azure Container Registry
Complete code for a REST API Docker container based on FastAPI in order to expose the model is also part of this repo
Helm Charts for deploying the model on a Kubernetes cluster
CICD pipelines for deploying a Python Azure Function, which acts as a Webhook listener

Coming soon!

Login with Azure CLI by running azure login. Select the desired subscription.
Go to the terraform/webhooks directory.
Create a copy of terraform.tfvars.example and copy it to terraform.tfvars.
Fill the correct values in terraform.tfvars.
Run terraform plan. You should see an overview of the resources that will be deployed.
If everything looks ok, run terraform deploy. This will create an MLflow Webhook on our Databricks Workspace, along with creating some secondary infrastructure.
Go to the terraform/serving directory and run steps 4-7 once again. This will deploy the infrastructure for serving our model.
On your Azure DevOps Project, create a Build pipeline and point it to cicd/model/azure-pipelines.yaml.
On your Azure DevOps Project, create a new Service Connection. For type, select Docker Registry. Select your Azure Subscription and the Azure Container Registry that was created with Terraform on step 8.
Create a variable for the Build Pipeline created on step 9 - name it registryServiceConnection and set the value to the service connection created on step 10
Create a variable for the Build Pipeline created on step 9 - name it acrRegistryName and set the value to the Azure Container Registry that was created by Terraform as part of step 8

On your Azure DevOps Project, create a Build pipeline and point it to cicd/functions/azure-pipelines.yaml.
Go to Library and create a Variable Group. Choose the option to link with an existing Azure Key Vault. Look for the keyvault that was deploying with Terraform and select it. Select the secrets from the keyvault, so that they are imported into the variable group.
Run the Build pipeline to deploy our Azure Function.

You should be good to go! Once you change the Model stage to Production, the MLflow Webhook will hopefully:

Make a post request to the Azure Function you deployed
This function will trigger the model build pipeline on Azure DevOps
The pipeline will:
- Download the model artifacts from MLflow Model Registry
- Scan the Dockerfile for vulnerabilities
- Build a Docker container
- Push this Docker container into Azure Container Registry

As a final step, you can deploy the container to AKS. To do so, from a terminal window:

Run make aks-install-cli to install Azure AKS Client and kubectl
In the Makefile, change RESOURCE_GROUP_NAME and AKS_CLUSTER_NAME accordingly. Then run make aks-login.
Finally, run make helm-install (change mymodel accordingly as well)

Please check the Issues. Feel free to contribute by solving them, or creating your first issue!