data-pipelines

There are 44 repositories under data-pipelines topic.

airflow
apache / airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration data-orchestrator data-pipelines data-science elt etl machine-learning mlops orchestration workflow-engine workflow-orchestration
Language:Python 34696
apache / dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
workflow-schedule azkaban airflow task-scheduler job-scheduler cloud-native data-pipelines orchestration workflow workflow-orchestration powerful-data-pipelines
Language:Java 12191
dagster-io / dagster
An orchestration platform for the development, production, and observation of data assets.
analytics dagster data-engineering data-integration data-orchestrator data-pipelines data-science etl metadata mlops orchestration python scheduler workflow workflow-automation
Language:Python 10367
infiniflow / ragflow
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
data-pipelines deep-learning document-parser document-understanding information-retrieval llm llmops machine-learning nlp ocr orchestration pdf-to-text preprocessing rag retrieval-augmented-generation table-structure-recognition
Language:Python 7540
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 7149
Unstructured-IO / unstructured
Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
deep-learning document-parsing machine-learning nlp ocr information-retrieval data-pipelines ml preprocessing pdf-to-text natural-language-processing pdf pdf-to-json document-image-analysis donut document-image-processing document-parser docx langchain llm
Language:HTML 6764
orchest / orchest
Build data pipelines, the easy way 🛠️
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker python data-pipelines orchest deployment kubernetes airflow dag etl etl-pipeline
Language:TypeScript 4021
fluvio
infinyon / fluvio
Lean and mean distributed stream processing system written in rust and web assembly.
cloud-native data-flow data-integration data-pipelines distributed-systems event-driven-architecture real-time rust serverless stateful stream-processing stream-processing-engine streaming streaming-data streaming-data-pipelines streaming-data-processing webassembly
Language:Rust 2682
elementary
elementary-data / elementary
The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
data-lineage data-governance data-warehouse snowflake bigquery data-analysis data-pipelines data-pipeline lineage data-reliability data-observability dataops dbt dbt-packages analytics-engineer dbt-artifacts redshift
Language:HTML 1745
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk connectors integration tap taps target targets loaders data-pipelines singer data-engineering
Language:Python 1611
combust / mleap
MLeap: Deploy ML Pipelines to Production
data-pipelines python scala scikit-learn spark tensorflow transformers
Language:Scala 1498
SciPhi-AI / R2R
The framework for fast development and deployment of RAG systems.
artificial-intelligence large-language-models retrieval retrieval-augmented-generation search chatbot data-pipelines deep-learning langchain llama-index llm machine-learning ocr pdf question-answering retrieval-systems
Language:Python 1243
odd-platform
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery data-lineage bigdata alerting lineage data-profiling data-exploration data-governance data-quality data-science data-observability
Language:Java 1122
data-engineering-wiki
data-engineering-community / data-engineering-wiki
The best place to learn data engineering. Built and maintained by the data engineering community.
data data-engineer data-engineering data-modeling data-pipelines database etl sql
Language:CSS 1041
dataform-co / dataform
Dataform is a framework for managing SQL based data operations in BigQuery
data-pipelines elt data-engineering business-intelligence analytics etl hacktoberfest
Language:TypeScript 794
raystack / optimus
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
airflow etl workflows automation golang bigquery data-warehouse analytics data-modelling analytics-engineering data-transformation data-pipelines elt business-intelligence dataops
Language:Go 737
artie-labs / transfer
Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database debezium elt redshift
Language:Go 539
versatile-data-kit
vmware / versatile-data-kit
One framework to develop, deploy and operate data workflows with Python and SQL.
data-science data-engineering sql trino data-lineage etl elt data-pipelines data-engineer data-warehouse warehouse analytics snowflake dataops data-engineering-pipeline python data pipeline data-structures database
Language:Python 412
fmind / mlops-python-package
Kickstart your MLOps initiative with a flexible, robust, and productive Python package.
automation data-pipelines data-science machine-learning mlflow mlops pandera pydantic python
Language:Jupyter Notebook 353
elementary-data / dbt-data-reliability
dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
data dbt-packages data-reliability data-pipeline-monitoring data-pipelines analytics analytics-engineering data-observability data-lineage dbt dbt-artifacts dbt-tests
Language:Python 345
recap
recap-build / recap
Work with your web service, database, and streaming schemas in a single format.
data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl
Language:Python 305
dataplane-app / dataplane
Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering workflow data-analysis data-integration dataplane datawarehouse etl pipelines robotics-process-automation rpa finance
Language:JavaScript 186
awesome-kubeflow
terrytangyuan / awesome-kubeflow
A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)
awesome awesome-list awesome-lists cloud-native cncf deep-learning devops kubeflow kubeflow-pipelines kubernetes machine-learning mlops workflow-orchestration data-pipelines data-science mle-pipelines
182
kevin-hanselman / dud
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
machine-learning dvcs data-science dataset mlops data-engineering data-pipelines
Language:Go 166
datajoint-python
datajoint / datajoint-python
Relational data pipelines for the science lab
datajoint scientific-computing databases data-analysis pipeline-framework python relational-databases relational-algebra relational-model mysql data-pipelines cloud-computing s3 workflow-management
Language:Python 163
koolreport / core
An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.
php reporting framework data-visualization data-viz data-analysis data-summarization reporting-engine reporting-tool mysql-reporting-tools php-reporting-tools php-reports data-pipelines data-pivot report-generator
Language:PHP 153
tuva-health / tuva
Main repo including core data model, data marts, reference data, terminology, and the clinical concept library
analytics-engineering bigquery data-analytics data-governance data-lineage data-pipelines data-warehouse dbt dbt-packages healthcare healthcare-analysis healthcare-data open-source redshift snowflake sql terminology
153
GoogleCloudPlatform / public-datasets-pipelines
Cloud-native, data onboarding architecture for Google Cloud Datasets
datasets data-pipelines airflow google-cloud cloud-composer open-data data-engineering data-architecture cloud-native bigquery cloud-storage
Language:Python 137
dataflint / spark
Performance Observability for Apache Spark
apache-spark big-data observability optimization data-pipeline data-pipelines databricks dataproc emr etl spark-operator
Language:TypeScript 127
patterns-app / patterns-devkit
Data pipelines from re-usable components
data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability data-pipeline data-pipelines etl-pipelines
Language:Python 106
smart-data-lake / smart-data-lake
Smart Automation Tool for building modern Data Lakes and Data Pipelines
data-lake scala spark smart-data-lake hadoop hive deltalake transform-data data-pipelines
Language:Scala 96
shravan-kuchkula / udacity-data-eng-proj-1
Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
airflow aws-redshift data-pipelines docker
Language:Python 88
confluentinc / learn-kafka-courses
Learn the basics of Apache Kafka® from leaders in the Kafka community with these video courses covering the Kafka ecosystem and hands-on exercises.
apache-kafka kafka kafka-streams data-pipelines data-pipeline ksqldb stream-processing
Language:Shell 83
beneath-hq / beneath
Beneath is a serverless real-time data platform ⚡️
dataops mlops data-science data-engineering data-pipelines developer-tools data-warehouse streaming etl analytics sql python go kubernetes beneath
Language:Go 81
DataCater / datacater
The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
cloud-native data-pipelines kafka kubernetes etl apache-kafka python
Language:JavaScript 81
linkedin / Hoptimator
Multi-hop declarative data pipelines
brooklin cdc data-pipelines flink kafka kafka-connect
Language:Java 77