etl-pipeline
There are 48 repositories under etl-pipeline topic.
- risingwave
risingwavelabs / risingwave
Streaming data platform. Real-time stream processing, low-latency serving, and Iceberg table management.
Language:Rust 8488 - unstract
Zipstack / unstract
No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
Language:Python 5923 - streampark
apache / streampark
Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
Language:Java 4225 - Language:TypeScript 4143
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Language:Jupyter Notebook 2303AlexIoannides / pyspark-example-project
Implementing best practices for PySpark ETL jobs and applications.
Language:Python 2019- Udacity-Data-Engineering-Projects
san089 / Udacity-Data-Engineering-Projects
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
Language:Python 1758 - goodreads_etl_pipeline
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
Language:Python 1430 Open-Source-Legal / OpenContracts
Enterprise-grade and API-first LLM workspace for unstructured documents, including data extraction, redaction, rights management, prompt playground, and more!
Language:TypeScript 949stitchfix / hamilton
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
Language:Python 860techascent / tech.ml.dataset
A Clojure high performance data processing system
Language:Clojure 727trustgraph-ai / trustgraph
Transform data into AI-ready context. Deploy knowledge graphs, private LLMs, and intelligent agents with complete data sovereignty. From data silos to actionable AI insights.
Language:Python 724SorellaLabs / brontes
A blazingly fast general purpose blockchain analytics engine specialized in systematic mev detection
Language:Rust 640- FlashLearn
Pravko-Solutions / FlashLearn
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
Language:Python 605 - Language:Scala 589
DataWithBaraa / sql-data-warehouse-project
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
Language:TSQL 403unbody-io / unbody
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
Language:TypeScript 355airscholar / e2e-data-engineering
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Language:Python 287ebonnal / streamable
concurrent & fluent interface for (async) iterables
Language:Python 285jitsucom / bulker
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
Language:Go 204jvalue / jayvee
Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Language:TypeScript 182- setl
SETL-Framework / setl
A simple Spark-powered ETL framework that just works 🍺
Language:Scala 182 data-engineering-community / data-engineering-project-template
This is a template you can use for your next data engineering portfolio project.
imsanjoykb / Data-Science-Regular-Bootcamp
Regular practice on Data Science, Machien Learning, Deep Learning, Solving ML Project problem, Analytical Issue. Regular boost up my knowledge. The goal is to help learner with learning resource on Data Science filed.
Language:Jupyter Notebook 128Wittline / uber-expenses-tracking
The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.
Language:Jupyter Notebook 121- Language:Rust 108
patterns-app / patterns-devkit
Data pipelines from re-usable components
Language:Python 107ContextData / VectorETL
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
Language:Python 104usc-isi-i2 / dig-etl-engine
Download DIG to run on your laptop or server.
- ApacheSpark
martandsingh / ApacheSpark
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
Language:Python 103 restarone / violet_rails
an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL
Language:Ruby 103sundios / SEO-Dashboard
SEO dashboard from Search console Data using the Google Search API, Mysql database , NodeJS RESTAPI( ExpressJS) and reactJs Dashboard
Language:JavaScript 94- Flowfile
Edwardvaneechoud / Flowfile
Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.
Language:Python 90 Indexical-Metrics-Measure-Advisory / watchmen-matryoshka-doll
Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
Language:Python 90runprism / prism
Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
Language:Python 86- onetlLanguage:Python 84
