Darwin1972 / azure-streaminganalytics-SQLdb-PowerBi-Grafana-tutorial

Step by step tutorial Microsoft Azure Streaming Analytics with SQL Database and two dashboards with on-prem Microsoft Power BI and Grafana.

azure-streaminganalytics-SQLdb-PowerBi-Grafana-tutorial

This tutorial is a simple step by step instruction to reach the following goals:

• Configure Microsoft Azure Streaming Analytics with Azure SQL
• Get the data into Microsoft Power BI and Grafana

Please note: This tutorial is only for dev / poc (proof of concept) environment and not for production environment.

Step 1

Prerequisite:

• Successful completion of the following tutorial: kafka-mirrormaker-azure-tutorial
• Power BI Desktop on your on-prem client pc
• Grafana on your on-prem client pc

Step 2

Create and configure Azure SQL:

Go to your Microsoft Azure subscription.

Create a resource

Create

Select SQL databases / Resource type: Single database.

Create

Configure:

• Resource gropu: kafkamirrormaker
• Database name: myStreamingDB

Server*: Create new

Enter server name (mysqlsever1968), server admin login and password:

OK

Configure database

Chose serverless and data max size 1GB:

Apply

Next: Networking >

Choose:

• Connectivity method: Public endpoint
• Allow Azure services and resources to access this server = yes
• Add current client IP address = yes

Next : Additional settings > 

Chose:

• Use existing data: Sample
• Enable Azure Defender for SQL: Not now

Review + create, Create

Go to resource

Step 3

Create and configure Azure Streaming Analytics:

Create

Enter job name: mysqlstreaming123

Next: Input >

Please make sure you have  tutorial completed the following tutoral: kafka-mirrormaker-azure-tutorial

Configure:

• Input type: Event Hub
• Event Hub namespace: kafkaazure
• Event Hub name: mymachine
• Event Hub policy name: Create new
• Event Hub consumer group: Use existing

Next: Output

Enter username and password from Step 2.

Validate:

Table: Create new

mymachine

Create

Enter the following value into the Kafka producer on Ubuntu 20.04:

{"sensor_id": 1,"ltime": 1613204245,"temp": 5.5,"status": 1}

{"sensor_id": 1,"ltime": 1613204245,"temp": 6.5,"status": 1}

{"sensor_id": 1,"ltime": 1613204245,"temp": 7.5,"status": 1}

Start streaming analytics job:

Start

Step 4

Validate streaming analytics job with SQL database query editor:

Query editor

Enter SQL server authentication login and password from Step 2.

OK

Enter the following value into the Kafka producer on Ubuntu 20.04:

{"sensor_id": 1,"ltime": 1613204245,"temp": 5.5,"status": 1}

{"sensor_id": 1,"ltime": 1614448017,"temp": 6.5,"status": 1}

{"sensor_id": 1,"ltime": 1614448093,"temp": 7.5,"status": 1}

Then run the following SQL statement: SELECT * FROM [dbo].[mymachine];

You should see the following results:

Step 5

Get the data into Microsoft Power BI:

Go to Overview:

Copy the server name: mysqlsever1968.database.windows.net for later use.

Start Power BI Desktop on your on-prem client pc.  

Get data and choose "More.."

Select Azure / Azure SQL database:

Connect

Enter server name: mysqlsever1968.database.windows.net

OK

Select Database and enter user name (server admin login) and password from Step 2.

Connect

Expand the myStreamingDB and select mymachine table.. 

Load

Select Data:

Step 6

Get the data into Grafana:

Login to Grafana and choose configuration / data sources:

Search for Microsoft SQL Server:

Select

Configure:

• Name: AzureDB
• Host: mysqlsever1968.database.windows.net
• Database: myStreamingDB
• User and password
• Encrypt: true

Save & Test

Select create / dashboard:

Add new panel

Select table as the visualization.

Enter the following SQL statement: SELECT * FROM [dbo].[mymachine] and change format as table:

Save

Save

Save dashboard: