data-orchestration
There are 5 repositories under data-orchestration topic.
- kestra
kestra-io / kestra
Orchestrate everything - from scripts to data, infra, AI, and business - as code, with UI and AI Copilot. Simple. Fast. Scalable.
Language:Java 22891 - Language:Java 7101
- Language:Go 5302
apache / incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
Language:C++ 310- Skytrax-Data-Warehouse
iam-mhaseeb / Skytrax-Data-Warehouse
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
Language:Python 139 Best practices for data workflows, integrations with the Modern Data Stack (MDS), Infrastructure as Code (IaC), Cloud Provider Services
Language:HCL 35jonathanneo / data-aware-orchestration
Data-aware orchestration with dagster, dbt, and airbyte
Language:Python 30ozkary / data-engineering-mta-turnstile
Data Engineering - Metropolitan Transportation Authority (MTA) Subway Data Analysis
Language:Jupyter Notebook 29SAP-samples / btp-data-to-value-workshop
This repo contains a dataset, exercises, and sample code for an end-to-end SAP BTP data-to-value bootcamp covering SAP HANA Cloud, SAP Data Warehouse Cloud, SAP Data Intelligence Cloud, and SAP Analytics Cloud.
Language:Jupyter Notebook 25astronomer / airflow-provider-fivetran-async
A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Language:Python 22dagster-io / dagster-quickstart
Get started with Dagster ASAP
Language:Python 14Alluxio / k8s-operator
An operator for managing Alluxio system on Kubernetes cluster
Language:Go 13anna-geller / kestra-ci-cd
CI/CD repository template to automate deployments of your production flows
Language:HCL 13stemitom / postgres-pipeline
A simple pipeline infrastructure with ETL pipeline contained in a Docker environment on Apache Airflow for orchestration and Postgres for data warehousing
Language:Python 7Bring Infrastructure as Code best practices to your data workflows with Kestra and Terraform
Language:HCL 5bravorod / mimic-ehr-pipeline
EHR pipeline that simulates MIMIC-IV patient data streams, performs advanced feature engineering and clinical severity scoring using machine learning (Random Forest Classifier), and prepares structured outputs for scalable downstream analytics
Language:Python 3longNguyen010203 / Finance-Data-Ingestion-Pipeline-with-Kafka
Develop a real-time data ingestion pipeline using Kafka and Spark. Collect minute-level stock data from Yahoo Finance, ingest it into Kafka, and process it with Spark Streaming, storing the results in Cassandra. Orchestrated the workflow using Airflow deployed on Docker.
Language:Python 3taquynhnga2001 / proptech-dagster
Build an ELT pipeline with dagster and dbt to schedule loading HDB resale transactions in Singapore into Google BigQuery data warehouse, then create Power BI dashboard to enhance insight exploration.
Language:Python 2zpencerguy / superdoppler
Data orchestration repo with Docker deployment
Language:Python 2- azure-data-factory-data-vault
Annielytix / azure-data-factory-data-vault
Working with SCD Type (Change Data Capture) and need a Data Vault model to test Azure Data Factory v2? - This Code with Help!
GADES-DATAENG / webinar
Code, scripts, and resources for the Data Engineering Fundamentals Course Webinar, covering Python, data pipelines, Apache Airflow, and more.
Language:Python 1jasontanx / prefect-learning
Prefect - Data orchestration tool practice & learning
Language:Python 1kingabzpro / 5-Airflow-Alternatives-for-Data-Orchestration-Tutorial
Code examples of Luigi, Prefect, Kedro, Dagster, and MageAI
Language:Python 1RafaelQSantos-RQS / airflow
This project provides a complete, containerized environment for deploying Apache Airflow, designed for both robust local development and production use. It uses a multi-stage docker-compose setup and a Makefile for a simplified and professional developer experience.
Language:Makefile 1- knowledge_graph_brain
ryandmonk / knowledge_graph_brain
MCP-native knowledge graph orchestrator that unifies data silos with GraphRAG, dynamic connectors, and local AI.
Language:TypeScript 1 - data-duck-pond
tanega / data-duck-pond
A poor-man's data lake fill with ducks
Language:Python 1 MostafaNabilll / end2end_pipeline
End to End data engineering project
Language:Python 0Wireforce-LLC / m3
☕ Data Orchestrator. Without abstractions
Language:TypeScript 0ddeutils / data-orchestra
❌ Full-Stack Data Orchestration config by Yaml template with Flask & HTMX
Language:Pythonhordiales / recommender_system_dagster
Curso MLOps ITBA. Orquestación data y modelo con Dagster. Airbyte. DBT. MLFlow
Language:Pythonjacquessham / airflow_notes
Repository to store scripts and notes on Airflow
Language:PythonLuk-kar / SmogSense
A modular data platform for end-to-end analytics, data pipeline orchestration, machine learning model registry in local open-source environments or for a commercial cloud setup.
Language:Jupyter Notebooknishthapant / airflow
This project orchestrates an end-to-end data pipeline for an e-commerce dataset using Apache Airflow (in Docker) and a separate dbt (data build tool) project. The pipeline transforms raw source data into structured, analytics-ready datasets.
Language:Pythonpedrohgoncalvess / drummond-dsl
dsl for drummond, a data pipeline orchestrator
Language:Scalaphiliporlando / dagster_university
I created this repo to follow along with the examples in the Dagster University Essentials course.
Language:Pythonvbluuiza / airflow-dbt-weather-pipeline
📄 Pipeline de dados meteorológicos utilizando Apache Airflow, dbt e PostgreSQL. Automatiza a ingestão, transformação e visualização de dados da API WeatherStack.
Language:Python
