Practices - Data engineering
Tools you need
Have a stackoverflow account : https://stackoverflow.com/
Have a github account : https://github.com/
A Scala IDE :
- Intellij Idea: https://www.jetbrains.com/fr-fr/idea/download
- Scala plugin : https://www.jetbrains.com/help/idea/discover-intellij-idea-for-scala.html
Docker and Compose
Take time to read and install
https://docs.docker.com/get-started/overview/
docker --version
Docker version 20.10.14 (example)
https://docs.docker.com/compose/
docker-compose --version
docker-compose version 1.29.2 (example)
Fork OR update the repo on your own Github account
Fork
Update your fork
TP - Functional programming for data engineering
TP - Functional programming for data engineering
TP - Batch with Spark and BI with metabase
TP - Getting started with Apache Kafka (Bash/Docker)
- Open tp-docker-kafka-bash
Apache Kafka - Scala Client - Condutkor
TP -- Open tp-docker-api
Apache Kafka - Spark Structured Streaming
TP -- Open tp-docker-api