Awesome Azure Databricks
Awesome content all about Azure Databricks features, integrations with Azure products, integrations with partner products and architecture guidance.
Azure Databricks Features
Delta Lake
- Real-Time Data Streaming With Databricks, Spark & Power BI - Bennie Haelen (Insight) - 03-03-2021
- From Kafka to Delta Lake using Apache Spark Structured Streaming Fabien Pomerol (Michelin) - 16-02-2021
Spark Structured Streaming
- Ingest Azure Event Hub Telemetry Data with Apache PySpark Structured Streaming on Databricks - Charles Chukwudozie (Microsoft) - 17-05-2021
- NYC Taxi - Cosmos DB & Structured streaming
- Real-Time Data Streaming With Databricks, Spark & Power BI - Bennie Haelen (Insight) - 03-03-2021
- How to Consume Data from Apache Kafka Topics and Schema Registry with Confluent and Azure Databricks - Caio Moreno (Microsoft) - 04-02-2021
- From Kafka to Delta Lake using Apache Spark Structured Streaming - Fabien Pomerol (Michelin) - 16-02-2021
- Stream Processing Event Hub Capture files with Autoloader - Raki Rahman (Microsoft) - 04-01-2021
- Exploring Azure Schema Registry with Spark - Raki Rahman (Microsoft) - 02-12-2020
- Incrementally Process Data Lake Files Using Azure Databricks Autoloader and Spark Structured Streaming API - Charles Chukwudozie (Microsoft) - 30-09-2020
- IBOR scenario using Azure Event Hubs and Azure Databricks - Laurent Bananier (Microsoft) - 17-08-2020
- Real time stream processing with Databricks and Azure Event Hubs - Xavier Sierra - 18-01-2019
Auto-Loader
- Stream Processing Event Hub Capture files with Autoloader - Raki Rahman (Microsoft) - 04-01-2021
- Incrementally Process Data Lake Files Using Azure Databricks Autoloader and Spark Structured Streaming API - Charles Chukwudozie (Microsoft) - 30-09-2020
DBFS (Databricks File System)
- Storage options and File manipulation Commands in Azure Databricks - Vikash Rajluhaniwal - 24-06-2021
Horovord
- Single-node and distributed Deep Learning on Databricks - Luuk van der Velden & Rik Jongerius (Codebeez) - 01-03-2021
Azure Product Integrations - Data & Analytics
ADLS gen2
- Storage options and File manipulation Commands in Azure Databricks - Vikash Rajluhaniwal - 24-06-2021
- How to connect Databricks to your Azure Data Lake - René Bremer (Microsoft) - 01-05-2021
- How to bring your modern data pipeline to production - René Bremer (Microsoft) - 25-10-2020
Power BI
- Real-Time Data Streaming With Databricks, Spark & Power BI - Bennie Haelen (Insight) - 03-03-2021
Azure Data Factory
- Putting the Factory in Azure Data Factory: Dynamically generated Pipelines - Daniel van der Ende (Go Data Driven) - 21-12-2021
- Just-in-time Azure Databricks access tokens and instance pools for Azure Data Factory pipelines using workspace automation - Nicholas Hurt (Microsoft) - 08-06-2020
- Single-node and distributed Deep Learning on Databricks - Luuk van der Velden & Rik Jongerius (Codebeez) - 01-03-2021
- How to bring your modern data pipeline to production - René Bremer (Microsoft) - 25-10-2020
Azure Cosmos DB
- NYC Taxi - Cosmos DB & Spark (Python)
- NYC Taxi - Cosmos DB & Structured streaming
- Using Spark 3 connector for Cosmos DB SQL API with Azure Databricks - Iranga Subasinghe (Microsoft) - 18-05-2021
- Cosmos DB & PySpark – Retrieve all attributes from all Collections under all Databases - Manoj Pandey (Microsoft) - 12-04-2021
- Using Python in Azure Databricks with Cosmos DB – DDL & DML operations by using “Azure-Cosmos” library for Python - Manoj Pandey (Microsoft) - 09-04-2021
- Connect to Cosmos DB from Databricks and read data by using Apache Spark to Azure Cosmos DB connector - Manoj Pandey (Microsoft) - 07-04-2021
- Azure CosmosDB and Databricks notebooks
- Integrate Azure Cosmos DB with Azure Databricks - Prashanth Xavier (Luxoft) - 09-01-2021
- How to bring your modern data pipeline to production - René Bremer (Microsoft) - 25-10-2020
Azure Event Hub
- Structured Streaming + Event Hubs Integration Guide for PySpark - Github
- Detecting SQL Column Decryption using Purview, Kafka, Kafdrop and Spark - Raki Rahman (Microsoft) - 03-10-2021
- Ingest Azure Event Hub Telemetry Data with Apache PySpark Structured Streaming on Databricks - Charles Chukwudozie (Microsoft) - 17-05-2021
- Real-Time Data Streaming With Databricks, Spark & Power BI - Bennie Haelen (Insight) - 03-03-2021
- Stream Processing Event Hub Capture files with Autoloader - Raki Rahman (Microsoft) - 04-01-2021
- Exploring Azure Schema Registry with Spark - Raki Rahman (Microsoft) - 02-12-2020
- IBOR scenario using Azure Event Hubs and Azure Databricks - Laurent Bananier (Microsoft) - 17-08-2020
- Real time stream processing with Databricks and Azure Event Hubs - Xavier Sierra (Fexco) - 18-01-2019
Azure Purview
- Databricks Notebook lineage using Azure Purview Atlas API - Vishal Anil (Microsoft) - 08-03-2021
- Data Lineage from Databricks to Azure Purview - Anders Boje Hertz & Oliver Rise Thomsen (Intellishore) - 16-03-2021
Microsoft Presidio
Azure Product Integrations - Platform & DevOps
Azure Networking
- How to connect Databricks to your Azure Data Lake - René Bremer (Microsoft) - 01-05-2021
Azure DevOps
- Nutter for testing of Databricks notebooks in the CI/CD pipeline - Alex Ott (Databricks) - 18-06-2021
- CI/CD Tenmplats for Databricks andAzure Devops
- Using Terraform Outputs Connect To Azure Databricks Workspace via Azure DevOps - Richie Lee (Sabin.io) - 28-05-2021
- Elegant CICD with Databricks notebooks - Rik Jongerius & Luuk van der Velden (Cloudbees) - 09-01-2021
- Databricks Notebook Promotion using Azure DevOps - Himansu Sekhar - 03-01-2021
- DataOps 3 - Databricks Code Promotion using DevOps CI/CD - YouTube - David Lusty (Microsoft) - 11-02-2020
- How to bring your modern data pipeline to production - René Bremer (Microsoft) - 25-10-2020
Azure Log Analytics
Azure Product Integrations - Data Science, ML & AI
Azure ML
- MLOps for Azure Databricks Example - Sascha Dittmann (Microsoft) - 13-04-2021
Architecture
- Launching Databricks at If - Valdas Maksimavičius (Microsoft Data Platform MVP) - 31-03-2021
- From Warehouse to Lakehouse – ELT/ETL Design Patterns with Azure Data Services - Mike Dobing (Microsoft) - 29-03-2021
- How to bring your modern data pipeline to production - René Bremer (Microsoft) - 25-10-2020
Partner Product Integrations
Confluent & Kafka
- From Kafka to Delta Lake using Apache Spark Structured Streaming Fabien Pomerol (Michelin) - 16-02-2021
- How to Consume Data from Apache Kafka Topics and Schema Registry with Confluent and Azure Databricks - Caio Moreno (Microsoft) - 04-02-2021
dbt
Terraform
- Using Terraform Outputs Connect To Azure Databricks Workspace via Azure DevOps - Richie Lee (Sabin.io) - 28-05-2021