pipeline

There are 93 repositories under pipeline topic.

serve
jina-ai / serve
☁️ Build multimodal AI applications with cloud-native stack
neural-search cloud-native deep-learning machine-learning framework grpc kubernetes multimodal mlops pipeline fastapi generative-ai docker jaeger llmops opentelemetry cncf microservice orchestration prometheus
Language:Python 21731
kestra
kestra-io / kestra
:zap: Universal Workflow Orchestration Platform — Code in any language, run anywhere. 900+ plugins for data, infrastructure, and AI automation.
orchestration data-orchestration high-availability infrastructure-as-code lowcode automation devops java low-code pipeline pipeline-as-code workflow hacktoberfest
Language:Java 21153
argoproj / argo-cd
Declarative Continuous Deployment for Kubernetes
argo kubernetes continuous-deployment gitops continuous-delivery docker cd cicd pipeline devops ci-cd argo-cd helm hacktoberfest jsonnet kustomize
Language:Go 20638
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
python workflow data-engineering data-science workflow-engine prefect infrastructure ml-ops data-ops automation orchestration data observability pipeline
Language:Python 20354
vector
vectordotdev / vector
A high-performance observability data pipeline.
events forwarder logs metrics observability parser pipeline router rust stream-processing vector
Language:Rust 20304
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration elt bigquery redshift snowflake data-pipeline mssql mysql postgresql s3 self-hosted
Language:Python 19514
taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines python taipy-gui workflow taipy-core hacktoberfest hacktoberfest2023 data-integration job-scheduler scenario scenario-analysis
Language:Python 18634
marimo-team / marimo
Transform data, train models, and run SQL with marimo — feels like a next-gen reactive notebook, stored as Git-friendly reproducible Python. Deploy as scripts, pipelines, endpoints, and apps. All from an AI-native editor (or your own).
notebooks python data-science machine-learning artificial-intelligence data-visualization developer-tools reactive pipeline web-app dag dataflow sql
Language:Python 15899
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 10741
kedro
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
pipeline kedro hacktoberfest mlops experiment-tracking python machine-learning machine-learning-engineering
Language:Python 10527
prql
PRQL / prql
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
data pipeline sql
Language:Rust 10469
projectdiscovery / httpx
httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
http bugbounty ssl-certificate pipeline pentest-tool cybersecurity osint hacktoberfest cli lib
Language:Go 8998
tektoncd / pipeline
A cloud-native Pipeline resource.
cdf hacktoberfest kubernetes pipeline tekton
Language:Go 8755
iam-veeramalla / Jenkins-Zero-To-Hero
Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.
argocd cicd docker gitops jenkins kubernetes pipeline
Language:Python 8682
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 8464
proposal-pipeline-operator
tc39 / proposal-pipeline-operator
A proposal for adding a useful pipe operator to JavaScript.
javascript operator pipeline proposal syntax tc39
Language:HTML 7784
brunch / brunch
:fork_and_knife: Web applications made easy. Since 2011.
javascript workflow build-automation pipeline brunch
Language:JavaScript 6780
nteract / papermill
📚 Parameterize, execute, and analyze notebooks
jupyter notebooks notebook-generator nteract publishing pipeline notebook python r julia scala
Language:Python 6262
GameDevMind
gonglei007 / GameDevMind
最全面的游戏开发技术图谱(Game Development Map)。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。
game-development unity3d 3d programming unreal-engine unrealengine game shader game-server framework mmorpg roadmap scrum pipeline game-framework cpp management devops gamedev unity
Language:Shell 5899
gaia-pipeline / gaia
Build powerful pipelines in any programming language.
go pipeline automation java python cplusplus build deployment kubernetes continuous-integration continuous-delivery continuous-testing devops devops-tools ruby distributed-systems gitops
Language:Go 5238
jx
jenkins-x / jx
Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
kubernetes cicd continuous-integration continuous-delivery accelerator openshift gitops environments pipeline devops golang tekton hacktoberfest
Language:Go 4666
cube-studio
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式
kubernetes inference mlops workflow ai pytorch spark argo kubeflow automl aihub gpt llmops notebook pipeline vgpu deepseek
Language:Python 4568
Production-Level-Deep-Learning
alirezadir / Production-Level-Deep-Learning
A guideline for building practical production-level deep learning systems to be deployed in real world applications.
machine-learning deep-learning pipeline scalable-applications production-system tfx kubeflow artificial-intelligence ai practical-machine-learning deployment system-design
4518
AutoRAG
Marker-Inc-Korea / AutoRAG
AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
analysis automl benchmarking document-parser embeddings evaluation llm llm-evaluation llm-ops open-source ops optimization pipeline python qa rag rag-evaluation retrieval-augmented-generation
Language:Python 4270
kubeflow / pipelines
Machine Learning Pipelines for Kubeflow
kubeflow-pipelines mlops kubeflow machine-learning kubernetes pipeline data-science
Language:Python 3940
towhee-io / towhee
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
machine-learning convolutional-networks embedding-vectors embeddings computer-vision image-processing video-processing feature-extraction image-retrieval unstructured-data feature-vector transformer milvus towhee vision-transformer vit pipeline llm
Language:Python 3422
firecow / gitlab-ci-local
Tired of pushing to test your .gitlab-ci.yml?
gitlab local pipeline git ci cd push untracked uncomitted gitlab-ci
Language:TypeScript 3312
nextflow
nextflow-io / nextflow
A DSL for data-driven computational pipelines
aws bioinformatics cloud dataflow docker groovy hello hpc nextflow pipeline pipeline-framework reproducible-research reproducible-science sge singularity singularity-containers slurm workflow-engine
Language:Groovy 3144
EpicGamesExt / BlenderTools
Blender addons that improve the game development workflow between Blender and Unreal.
python blender blender-addon unreal pipeline
Language:Python 2928
apptension / saas-boilerplate
SaaS Boilerplate - Open Source and free SaaS stack that lets you build SaaS products faster in React, Django and AWS. Focus on essential business logic instead of coding repeatable features!
cdk stack boilerplate frontend backend workers pipeline django docker saas serverless python react typescript vitejs aws business product aws-esc hacktoberfest
Language:TypeScript 2624
alibaba / pipcook
Machine learning platform for Web developers
js machine-learning pipeline tensorflow
Language:TypeScript 2580
EntilZha / PyFunctional
Python library for creating data pipelines with chain functional programming
python datascience pipeline data functional-programming
Language:Python 2470
InternLM / HuixiangDou
HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance
chatbot llm rag dsl lark pipeline robot wechat application multimodal image-retrieval assistant assistant-chat-bots group-chat
Language:Python 2430
instill-core
instill-ai / instill-core
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
unstructured-data low-code developer-tools etl no-code open-source hacktoberfest ai api cli generative-ai golang gpt llm pipeline python stable-diffusion typescript
Language:Python 2284
go-streams
reugn / go-streams
A lightweight stream processing library for Go
stream-processing pipeline etl kafka data-stream kafka-streams streams redis pulsar data-pipeline streaming-data stream-processor aerospike websocket nats-streaming streaming-api windowing throttling low-code workflow
Language:Go 2096
mara / mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
etl data-integration python postgresql pipeline data
Language:Python 2082