felipeall / spotify-charts-pipeline

Get the daily Spotify Charts data history and load to a PostgreSQL database

Spotify Charts Pipeline

Get the daily Spotify Charts data history and load to a PostgreSLQ database

Set up

1. Register on Spotify Developer
2. Create an app
3. Add a Redirect URI to the app (e.g. http://localhost:9000)

Poetry

Poetry is a python dependency management tool to manage dependencies, packages, and libraries in your python project. Install it by following the official documentation

Running

Clone the repository

git clone https://github.com/felipeall/spotify-charts-pipeline.git

Access the project root folder

cd spotify-charts-pipeline

Create a .env file and add your CLIENT_ID, SECRET_ID and REDIRECT_URI

cp .env.template .env

Create a Poetry virtual environment

poetry shell

Install the required packages

poetry install

Instantiate the PostgreSQL docker container

docker compose -p spotify_charts up -d --build

Run the pipeline with the desired arguments

python src/main.py [from_date] [to_date] [country_code]

Example

python src/main.py --from_date 2023-01-01 --to_date 2023-01-31 --country_code BR

Unit tests

docker compose -f docker-compose.tests.yml up --build --abort-on-container-exit --remove-orphans ; docker compose -f docker-compose.tests.yml rm -fsv