The purpose of the script is to automate the deployment process for spinning up demo environment to host CMS data in Synapse Analytics. The script takes care of deploying Resources in Azure like Synapse Analytics, Azure Data Factory and Azure Data Lake Storage. It also downloads Healthcare based CMS data and loads into Synapse database. This CMS data is then used for reporting using Power BI.
The Main script works in following steps:
- Prompts User for Resource Group Name and Password
- Presets various variables like SQL Server Name, Azure Data Factory Name, Storage Name, etc. using Resource Group Name provided in Step 1.
- Creates SQL Server
- Creates Synaspe (SQL Pool)
- Gets your Public IP so it can configured in the Synapse Firewall
- Sets the Firewall Rules for Synapse
- Creates Azure Data Factory
- Creates Azure Data Lake Storage Account
- Creates Storage Containers for the CMS data
- Creates parameters file for the Azure Data Factory ARM Template
- Deploys Azure Data Factory ARM Template that contains,pipelines, datasets and dataflows
- Gets Storage Access Key
- Gets Connection String for the Synapse Pool
- Creates Tables and Views in Synapse using script synapseCMSddls.sql
- Downloads CMS PartD data from website https://www.cms.gov/ and saves into the Storage
- Executes Azure Data Factory pipelines that reads CMS Data from Storage and loads into SQL Pool (Synapse)
- Next step is to get the Power BI tempate and connect to the Synapse instance.
Pre-requisites:
- Install Powershell version 7.0.3 or later (Reference Link: https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7)
- Install Azure PowerShell version 4.7.0 or later (Reference Link: https://docs.microsoft.com/en-us/powershell/azure/install-az-ps?view=azps-4.7.0)
- Install Powershell Module SQLServer (Reference Link: https://docs.microsoft.com/en-us/sql/powershell/download-sql-server-ps-module?view=sql-server-ver15)
- Access to the Azure Portal to be able to create resources
How to use the script:
- Clone Git repository local
- Open Powershell and navigate to the git folder
- Execute the script deploySynapseStorageADF.ps1 from Powershell, example
./deploySynapseStorageADF.ps1
Things to note:
- In order to connect to Synapse instance, make sure to add your IP address to the SQLPool Firewall settings
- You might have to also enable flag in SQLPool Firewall settings to be able to allow Polybase "Allow Azure services and resources to access this server"
- You might have to regenerate Master Key Password in Synapse
ALTER MASTER KEY REGENERATE WITH ENCRYPTION BY PASSWORD = '<Password>'". Although synapseCMSddls.sql does reset the Master Key.