airflow-dag

There are 0 repository under airflow-dag topic.

goodreads_etl_pipeline
san089 / goodreads_etl_pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3 warehouse data-lake scheduler data-migration data-engineering data-engineering-pipeline python goodreads-data-pipeline airflow-dag etl-job
Language:Python 1237
allegro / bigflow
A Python framework for data processing on GCP.
bigquery beam airflow-dag python-framework gcp workflows python composer dag dataproc dataflow
Language:Python 116
elyra-ai / airflow-notebook
This repository is no longer maintained.
airflow airflow-dag jupyter-notebook
Language:Python 15
chayansraj / Python-ETL-pipeline-using-Airflow-on-AWS
This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
airflow airflow-dag automation aws aws-ec2 dags data-engineering etl-pipeline python3 ssh-server virtual-machine visual-studio-code artificial-intelligence data database elt reverse-etl transformation
Language:Python 13
airflow-ditto
angadsingh / airflow-ditto
An airflow DAG transformation framework
airflow airflow-dag graph-algorithms networkx aws azure hdinsight emr spark livy yarn framework extensible isomorphism dataflow graph-manipulation
Language:Python 5
k0rsakov / dag_factory
Фабрика DAG
airflow airflow-dag airflow-dags apache-airflow dag-factory data data-engineering
Language:Python 2
minhky2185 / incident_data_warehouse
Build a data warehouse from scratch, including full load, daily incremental load, design schema, SCD Type 1 and 2.
airflow bridge-table data-engineering data-warehouse docker etl-pipeline orchestration powerbi python redshift s3 scd-type-2 staging-area star-schema incident-data-warehouse airflow-dag airflow-plugin etl-automation scd-type-1
Language:Python 2
IgnacioNMiranda / trigger-dag-run-params-demo
Apache Airflow demo project that setup 3 DAGs to explain how to pass parameters from a DAG to a triggered DAG.
airflow-dag airflow-dags airflow-operator airflow-operators apache-airflow python python3
Language:Python 1
vicmar57 / COVID-19-Live-Tweet-Analyzer-Kafka-Spark-Cassandra-and-Airflow
Analysing live tweets from twitter by generating a big data pipeline and scheduling it with Airflow (Using also Kafka for tweet ingestion, Cassandra for storing parsed tweets, and Spark for Analysis)
kafka spark cassandra-database cassandra-cql python3 python-3 airflow-dag airflow
Language:Python 1
adrianmarino / datitos
Datitos - TP2 with steroids
pytorch deep-learning classification python optuna apache-airflow airflow-dag
Language:Jupyter Notebook 0
akashkatakam / INFO-7374-production-data-pipelines
A dashboard with sentiment scores of the tweets
python3 airflow-dag aws-lambda sentiment-analysis object-detection microservices
Language:Python 0
djamier / docker-airflow
Orchestrate data pipeline using airflow
airflow airflow-docker airflow-dag docker-airflow python
Language:Python 0
judeleonard / e-commerce_activity_tracking
This is an ELT data pipeline setup to track the activities of an e-commerce website based on orders, reviews, deliveries and shipment date. This project utilized technologies like Airflow, AWS RDS-Postgres, Python etc.
dataengineering datapipeline e-commerce elt-pipeline airflow airflow-dag airflow-docker postgres
Language:Python 0
marigroc / pinterest-data-pipeline53
Creation of the almost-real time data processing pipeline for the Pintrest posts.
airflow-dag api-gateway aws databricks-notebooks e2c iam-role kafka-streams kinesis-stream
Language:Jupyter Notebook 0
pratiksha2712 / ETL-Pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
airflow airflow-dag apache-spark data-engineering data-lake data-migration etl-framework etl-pipeline livy python redshift s3 scheduler spark warehouse
Language:Python 0
RangaAmirapu / DataEngineeringProjects
Data Engineering Projects on data modelling, data warehousing, data lake development, orchestration and analysis
data-engineering redshift postgresql-database cassandra-database data-warehousing data-lake boto3 airflow airflow-dag
Language:Jupyter Notebook 0
lepiaf / airflow-training
Small project to play around Apache Airflow and ETL
airflow airflow-dag
Language:Jupyter Notebook