data-pipelines

There are 64 repositories under data-pipelines topic.

• pathway

  pathwaycom / pathway

  Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

  batch-processingkafkapathwaypythonstreamingmachine-learning-algorithmsreal-timedata-analyticsdata-pipelinesdata-processingdataflowetletl-frameworkiot-analyticsruststream-processingtime-series-analysis
  Language:Python 49336
• airflow

  apache / airflow

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  airflowapacheapache-airflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationpythonschedulerworkflowworkflow-engineworkflow-orchestration
  Language:Python 43114

• dagster-io / dagster

  An orchestration platform for the development, production, and observation of data assets.

  analyticsdagsterdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceetlhacktoberfestmetadatamlopsorchestrationpythonschedulerworkflowworkflow-automation
  Language:Python 14363

• apache / dolphinscheduler

  Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

  workflow-scheduleazkabanairflowtask-schedulerjob-schedulercloud-nativedata-pipelinesorchestrationworkflowworkflow-orchestrationpowerful-data-pipelines
  Language:Java 13930

• Unstructured-IO / unstructured

  Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

  data-pipelinesdeep-learningdocument-image-analysisdocument-image-processingdocument-parserdocument-parsingdocxdonutinformation-retrievallangchainllmmachine-learningmlnatural-language-processingnlpocrpdfpdf-to-jsonpdf-to-textpreprocessing
  Language:HTML 13137

• mage-ai / mage-ai

  🧙 Build, run, and manage data pipelines for integrating and transforming data.

  artificial-intelligencedatadata-engineeringdata-integrationdata-pipelinesdata-sciencedbteltetlmachine-learningorchestrationpipelinepipelinespythonreverse-etlsparksqltransformation
  Language:Python 8517
• fluvio

  infinyon / fluvio

  🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

  cloud-nativedata-analyticsdata-flowdata-integrationdata-pipelinesdistributed-systemsevent-driven-architecturereal-timerustserverlessstatefulstream-processingstream-processing-enginestreamingstreaming-analyticsstreaming-datastreaming-data-pipelinesstreaming-data-processingwebassembly
  Language:Rust 5075
• preswald

  StructuredLabs / preswald

  Preswald is a WASM packager for Python-based interactive data apps: bundle full complex data workflows, particularly visualizations, into single files, runnable completely in-browser, using Pyodide, DuckDB, Pandas, and Plotly, Matplotlib, etc. Build dashboards, reports, and notebooks that run offline, load fast, and share like a document.

  aianalyticsanalytics-engineeringcopilotdatadata-applicationsdata-infrastructuredata-pipelinesdata-sdkdata-visualizationgptllmopen-sourcepythonschema-managementvscode
  Language:Python 4299

• orchest / orchest

  Build data pipelines, the easy way 🛠️

  data-sciencemachine-learningpipelinesidejupytercloudself-hostedjupyterlabnotebooksdockerpythondata-pipelinesorchestdeploymentkubernetesairflowdagetletl-pipeline
  Language:TypeScript 4143

• Netflix / maestro

  Maestro: Netflix’s Workflow Orchestrator

  analyticsautomationbatch-processingdagdata-engineeringdata-opsdata-orchestratordata-pipelinesdata-scienceeltetljavamachine-learningmaestromlopsorchestrationschedulerworkflowworkflow-engineworkflow-orchestration
  Language:Java 3642

• ucbepic / docetl

  A system for agentic LLM-powered data processing and ETL

  dataetlllmpythondata-pipelineseltworkflowagentssemantic-datadocument-processingunstructured-dataunstructured-data-analysisdocument-analysis
  Language:Python 3034

• meltano / meltano

  Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

  dataopsdataops-platformeltopen-sourceopensourcedatapipelinesextract-datameltanomeltano-sdkconnectorsintegrationtaptapstargettargetsloadersdata-pipelinessingerdata-engineering
  Language:Python 2248
• elementary

  elementary-data / elementary

  The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

  data-lineagedata-governancedata-warehousesnowflakebigquerydata-analysisdata-pipelinesdata-pipelinelineagedata-reliabilitydata-observabilitydataopsdbtdbt-packagesanalytics-engineerdbt-artifactsredshift
  Language:HTML 2181
• data-engineering-wiki

  data-engineering-community / data-engineering-wiki

  The best place to learn data engineering. Built and maintained by the data engineering community.

  datadatabasedata-engineeringdata-engineersqldata-modelingdata-pipelinesetl
  Language:CSS 1810
• feldera

  feldera / feldera

  The Feldera Incremental Computation Engine

  data-analyticsdata-pipelinesdatabaseincremental-computationincremental-view-maintenanceivmmaterialized-viewsreal-timerustsqlstreaming
  Language:Rust 1673

• yobix-ai / extractous

  Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.

  data-pipelinesdocxetletl-pipelinesextractionllmmachine-learningnatural-language-processingnlpocrpdfpdf-parserragrusttikaunstructuredunstructured-data
  Language:Rust 1618

• combust / mleap

  MLeap: Deploy ML Pipelines to Production

  scikit-learnsparkdata-pipelinestransformerstensorflowscalapython
  Language:Scala 1528

• pyper-dev / pyper

  Concurrent Python made simple

  asyncioconcurrencypythonthreadingdata-pipelinesdata-processingmultiprocessingparallel-computingdatadata-collectiondata-engineering
  Language:Python 1506

• OpenDCAI / DataFlow

  Easy Data Preparation with latest LLMs-based Operators and Pipelines.

  datadata-cleaningdata-pipelinesdata-processingdata-sciencedata-synthesisllmsoperatorsdata-agentsglang-bankendvllm-backendgradio-interfacequick-data-processing
  Language:Python 1446

• fmind / mlops-python-package

  Kickstart your MLOps initiative with a flexible, robust, and productive Python package.

  mlopspythonautomationdata-pipelinesdata-sciencemachine-learningmlflowpanderapydantic
  Language:Jupyter Notebook 1367
• odd-platform

  opendatadiscovery / odd-platform

  First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

  ossdata-platformmetadatametadata-managementdata-pipelinesdata-engineeringobservabilitydata-catalogdatacatalogdata-discoverydata-lineagebigdataalertinglineagedata-profilingdata-explorationdata-governancedata-qualitydata-sciencedata-observability
  Language:Java 1365

• bruin-data / bruin

  Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

  analyticsbigquerydata-analysisdata-ingestiondata-modelingdata-pipelinesdata-platformdata-transformationpythonsnowflakesql
  Language:Go 1225

• amphi-ai / amphi-etl

  Visual Data Preparation and Transformation. Low-Code Python-based ETL.

  datadata-pipelinesetlstructured-dataunstructured-datadata-analysisdata-sciencedatatransformationanalytics-automationdata-preparation
  Language:JavaScript 1104

• dataform-co / dataform

  Dataform is a framework for managing SQL based data operations in BigQuery

  data-pipelineseltdata-engineeringbusiness-intelligenceanalyticsetlhacktoberfest
  Language:TypeScript 935

• raystack / optimus

  Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

  airflowetlworkflowsautomationgolangbigquerydata-warehouseanalyticsdata-modellinganalytics-engineeringdata-transformationdata-pipelineseltbusiness-intelligencedataops
  Language:Go 754

• artie-labs / transfer

  Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.

  snowflakecdcchange-data-capturegolangbigquerykafkaapache-kafkadata-integrationdata-pipelinesdatabasedebeziumeltredshift
  Language:Go 680

• elementary-data / dbt-data-reliability

  dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

  datadbt-packagesdata-reliabilitydata-pipeline-monitoringdata-pipelinesanalyticsanalytics-engineeringdata-observabilitydata-lineagedbtdbt-artifactsdbt-tests
  Language:Python 468
• versatile-data-kit

  vmware / versatile-data-kit

  One framework to develop, deploy and operate data workflows with Python and SQL.

  data-sciencedata-engineeringsqltrinodata-lineageetleltdata-pipelinesdata-engineerdata-warehousewarehouseanalyticssnowflakedataopsdata-engineering-pipelinepythondatapipelinedata-structuresdatabase
  Language:Python 468

• dataplane-app / dataplane

  Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

  golangairflowkubernetesdataschedulerworkflowsworkflow-automationdata-pipelinesdata-sciencedata-engineeringworkflowdata-analysisdata-integrationdataplanedatawarehouseetlpipelinesrobotics-process-automationrpafinance
  Language:JavaScript 356
• recap

  gabledata / recap

  Work with your web service, database, and streaming schemas in a single format.

  data-catalogmetadatadata-discoveryrecapdata-engineeringdata-integrationdata-pipelinesetl
  Language:Python 343

• dataflint / spark

  Drop-in replacement for Apache Spark UI

  apache-sparkbig-dataobservabilityoptimizationdata-pipelinedata-pipelinesdatabricksdataprocemretlspark-operator
  Language:TypeScript 340

• tuva-health / tuva

  Main repo including core data model, data marts, data quality tests, and terminology sets.

  analytics-engineeringbigquerydata-analyticsdata-governancedata-lineagedata-pipelinesdata-warehousedbtdbt-packageshealthcarehealthcare-analysishealthcare-dataopen-sourceredshiftsnowflakesqlterminology
  Language:HTML 276
• awesome-kubeflow

  terrytangyuan / awesome-kubeflow

  A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)

  awesomeawesome-listawesome-listscloud-nativecncfdeep-learningdevopskubeflowkubeflow-pipelineskubernetesmachine-learningmlopsworkflow-orchestrationdata-pipelinesdata-sciencemle-pipelines
  217

• kevin-hanselman / dud

  A lightweight CLI tool for versioning data alongside source code and building data pipelines.

  machine-learningdvcsdata-sciencedatasetmlopsdata-engineeringdata-pipelines
  Language:Go 213
• datajoint-python

  datajoint / datajoint-python

  Relational data pipelines for the science lab

  datajointscientific-computingdatabasesdata-analysispipeline-frameworkpythonrelational-databasesrelational-algebrarelational-modelmysqldata-pipelinescloud-computings3workflow-management
  Language:Python 184

• koolreport / core

  An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.

  phpreportingframeworkdata-visualizationdata-vizdata-analysisdata-summarizationreporting-enginereporting-toolmysql-reporting-toolsphp-reporting-toolsphp-reportsdata-pipelinesdata-pivotreport-generator
  Language:PHP 176