zuzhi / practical-data-engineering

Real estate dagster pipeline

Practical Data Engineering Project

This is a practical example of a data engineering project with real-estates. The connected blog post about Building a Data Engineering Project in 20 Minutes you can find on my website. Topics are:

• Getting the Data – Scraping with BeautifulSoup
• Storing on S3-MinIO
• Custom Change Data Capture (CDC)
• Adding Database features to S3 – Delta Lake & Spark
• Machine Learning part – Jupyter Notebook
• Ingesting Data Warehouse for low latency – Apache Druid
• The UI with Dashboards and more – Apache Superset
• Orchestrating everything together – Dagster
• DevOps engine – Kubernetes

The Status of the project you find here.

Starting Dagster

To get MinIO, Spark, Kubernetes, etc. ready, check the representive folder in here.

1. MinIO started
2. Kubernetes ready
3. Spark image and role and namespaces ready
4. cd src/pipelines/real-estate and start dagit with dagit

Start issue

1. ImportError

ImportError: cannot import name 'SubscriptionObserver' from 'graphql_ws.gevent'

Manually install graphql_ws with version 0.3.1.

pip install graphql_ws==0.3.1

2. dagster.check.CheckError

dagster.check.CheckError: Invariant failed. Description: Attempted to deserialize class "InstigatorState" which is not in the whitelist.

Use a clean dagster home.

export DAGSTER_HOME="~/dagster-home-real-estate"

3. ModuleNotFoundError

ModuleNotFoundError: No module named 'bs4'

Install bs4

pip install bs4

4. UserWarning

UserWarning: No dagster instance configuration file (dagster.yaml) found at /Users/zuzhi/dagster-home-real-estate.

Create an empty dagster.yaml file in /Users/zuzhi/dagster-home-real-estate.

touch ~/dagster-home-real-estate/dagster.yaml