da-199 / news-headlines

Pipeline that extracts data from sites for a dashboard.

Data Sources:

• New York Times Homepage API
• NBC News front page
• ABC News front page

ETL Architecture Overview:

1. Files in the sources folder are scheduled to extract data on an hourly basis at the start of the hour.
2. Transformations on tables are performed.
3. generate_queries.py turns the dataframe into SQL queries.
4. database_connection.py loads the data to PostgreSQL by running the queries.

Data Model

Google Data Studio Visualization