tanayseven / movie-data-engieering

The Repo for my talk on Intro to DBT

DBT Movie Recommendation System

Setup

export PYTHONPATH=$PWD # in project root

cd alembic
python populate_alembic.py

Alembic commands

alembic history

alembic revision --autogenerate -m "initial tables"

alembic upgrade head

alembic downgrade base

DBT commands

dbt run

dbt seed

dbt test

dbt docs generate && dbt docs serve

Airflow on local commands

Run Airflow in Docker compose

docker-compose up

Get the admin password for Airflow

docker exec -it dbt-intro-airflow-1 cat standalone_admin_password.txt && echo

Copy DBT project to DAGs folder

cp -r ./dbt_intro/ ./airflow/dags/

Run recommendation engine and generate recommendations

python -m recommendation_system.train # make sure the table is deleted before running this