Giters

rd-coutinho / ELT_Twitter_API

Data engineering project about an ELT process using Apache Airflow and Apache Spark.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

airflowapietl-pipelinedatalakepysparksparkdata-engineering

ELT_Twitter_API

Repository for all the scripts about a data engineering project involving an ELT process extracting data from the Twitter API (querying for tweets referencing @AluraOnline) using Apache Airflow (v2.3.2), loading to a datalake and transforming the data to make it structured using Apache Spark (PySpark).

About

Data engineering project about an ELT process using Apache Airflow and Apache Spark.

airflowapietl-pipelinedatalakepysparksparkdata-engineering

Languages

Language:Python 100.0%