zhangdi-devel / de_insight

Data engineering project at Insight

Home Page:https://zhangdi-devel.github.io/de_insight

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Dashboard for online games

Project idea

Build a real-time anomaly detection pipeline and a dashboard to show the aggregated results for users.

What is the purpose, and most common use cases?

Purpose:
  • real-time anomaly detection based on user reporting and in-game aggregated stats
  • dashboard that shows the most reported users
Use cases:
  • detect game cheaters

Which technologies are well-suited?

Datasets:
  • PUBG match deaths and statistics
  • Simulated reporting events
Technologies:
  • Kafka
  • Flink
  • Postgres
  • Dash

What are the engineering challenges

  • Stateful joining in Flink

Proposed architechture

Kafka -> Flink -> Postgres -> Dash

About

Data engineering project at Insight

https://zhangdi-devel.github.io/de_insight

License:MIT License


Languages

Language:Scala 86.3%Language:Python 11.1%Language:Shell 2.6%