nasimmehrshid / Synapse-hackathon-28-29

Synapse Hackathon

Exercise 0: Env setup (D1 30 min)

Portal or ARM deploy

End-To-End Solution Architecture:

Exercise 1: Load data (D1 1H)

At this exercise you should connect to different data sources that WWI needs to collect data and create a pipeline to move data into a centralized data lake. No transformation over data is required. The solution should cover delta load for big tables.

Exercise 2: Transformation (Raw to CDM) (D1 4H)

• Basic data quiality check (null value, strings, special charecters, bad data types)
• Database templates (.parquet)
  • Mapping tool
  • Transformation (build by mapping tool)
  • Data Flows / pipelines (automated)

Exercise 3: Datawarehouse (D2 3H)

Star schema transformation:

• Data flows
• Spark Pools

Save data as Delta files

Exercise 4: Reporting (Mark-Marijn) (D2 1H)

PowerBI integration with Synapse Push data marts to PowerBI

Good to have Exercises:

Exercise 5: Machine Learning (share a ONNX model to play with)

Exercise 6: Purview

Exercise 7: Data Mesh