pipeline

There are 104 repositories under pipeline topic.

kestra
kestra-io / kestra
Orchestrate everything - from scripts to data, infra, AI, and business - as code, with UI and AI Copilot. Simple. Fast. Scalable.
automation data-orchestration devops hacktoberfest high-availability infrastructure-as-code java low-code lowcode orchestration pipeline pipeline-as-code workflow
Language:Java 22891
serve
jina-ai / serve
☁️ Build multimodal AI applications with cloud-native stack
cloud-native cncf deep-learning docker fastapi framework generative-ai grpc jaeger kubernetes llmops machine-learning microservice mlops multimodal neural-search opentelemetry orchestration pipeline prometheus
Language:Python 21779
argoproj / argo-cd
Declarative Continuous Deployment for Kubernetes
argo kubernetes continuous-deployment gitops continuous-delivery docker cd cicd pipeline devops ci-cd argo-cd helm hacktoberfest jsonnet kustomize
Language:Go 21147
PrefectHQ / prefect
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine
Language:Python 20772
vector
vectordotdev / vector
A high-performance observability data pipeline.
router logs metrics rust observability forwarder vector parser events stream-processing pipeline hacktoberfest
Language:Rust 20648
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
bigquery change-data-capture data data-analysis data-collection data-engineering data-integration data-pipeline elt etl java mssql mysql pipeline postgresql python redshift s3 self-hosted snowflake
Language:Python 19980
taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 18849
marimo-team / marimo
A reactive notebook for Python — run reproducible experiments, query with SQL, execute as a script, deploy as an app, and version with git. Stored as pure Python. All in a modern, AI-native editor.
artificial-intelligence dag data-science data-visualization dataflow developer-tools machine-learning notebooks pipeline python reactive sql web-app
Language:Python 16966
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 10903
kedro
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
experiment-tracking hacktoberfest kedro machine-learning machine-learning-engineering mlops pipeline python
Language:Python 10624
prql
PRQL / prql
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
data pipeline sql
Language:Rust 10507
projectdiscovery / httpx
httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
bugbounty cli cybersecurity hacktoberfest http lib osint pentest-tool pipeline ssl-certificate
Language:Go 9151
iam-veeramalla / Jenkins-Zero-To-Hero
Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.
argocd cicd docker gitops jenkins kubernetes pipeline
Language:Python 8910
tektoncd / pipeline
A cloud-native Pipeline resource.
cdf hacktoberfest kubernetes pipeline tekton
Language:Go 8793
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 8517
proposal-pipeline-operator
tc39 / proposal-pipeline-operator
A proposal for adding a useful pipe operator to JavaScript.
javascript operator pipeline proposal syntax tc39
Language:HTML 7804
brunch / brunch
🍴 Web applications made easy. Since 2011.
brunch build-automation javascript pipeline workflow
Language:JavaScript 6774
nteract / papermill
📚 Parameterize, execute, and analyze notebooks
jupyter notebooks notebook-generator nteract publishing pipeline notebook python r julia scala
Language:Python 6304
GameDevMind
gonglei007 / GameDevMind
最全面的游戏开发技术图谱(Game Development Map)。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。
3d cpp devops framework game game-development game-framework game-server gamedev management mmorpg pipeline programming roadmap scrum shader unity unity3d unreal-engine unrealengine
Language:Shell 6002
gaia-pipeline / gaia
Build powerful pipelines in any programming language.
go pipeline automation java python cplusplus build deployment kubernetes continuous-integration continuous-delivery continuous-testing devops devops-tools ruby distributed-systems gitops
Language:Go 5234
jx
jenkins-x / jx
Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
accelerator cicd continuous-delivery continuous-integration devops environments gitops golang hacktoberfest kubernetes openshift pipeline tekton
Language:Go 4671
cube-studio
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习/大模型AI平台，mlops算法链路全流程，算力租赁平台，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU虚拟化，边缘计算，标注平台自动化标注，deepseek等大模型sft微调/奖励模型/强化学习训练，vllm/ollama/mindie大模型多机推理，私有知识库，AI模型市场，支持国产cpu/gpu/npu 昇腾生态，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式
ai aihub argo automl deepseek gpt inference kubeflow kubernetes llmops mlops notebook pipeline pytorch spark vgpu workflow
Language:Python 4668
Production-Level-Deep-Learning
alirezadir / Production-Level-Deep-Learning
A guideline for building practical production-level deep learning systems to be deployed in real world applications.
ai artificial-intelligence deep-learning deployment kubeflow machine-learning pipeline practical-machine-learning production-system scalable-applications system-design tfx
4527
AutoRAG
Marker-Inc-Korea / AutoRAG
AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation
analysis automl benchmarking document-parser embeddings evaluation llm llm-evaluation llm-ops open-source ops optimization pipeline python qa rag rag-evaluation retrieval-augmented-generation
Language:Python 4388
kubeflow / pipelines
Machine Learning Pipelines for Kubeflow
kubeflow-pipelines mlops kubeflow machine-learning kubernetes pipeline data-science
Language:Python 3985
firecow / gitlab-ci-local
Tired of pushing to test your .gitlab-ci.yml?
gitlab local pipeline git ci cd push untracked uncomitted gitlab-ci
Language:TypeScript 3443
towhee-io / towhee
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
computer-vision convolutional-networks embedding-vectors embeddings feature-extraction feature-vector image-processing image-retrieval llm machine-learning milvus pipeline towhee transformer unstructured-data video-processing vision-transformer vit
Language:Python 3429
cocoindex
cocoindex-io / cocoindex
Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!
ai change-data-capture data data-indexing etl indexing pipeline python rag real-time rust semantic-search data-engineering data-infrastructure data-processing help-wanted knowledge-graph llm context-engineering hacktoberfest
Language:Rust 3202
nextflow
nextflow-io / nextflow
A DSL for data-driven computational pipelines
bioinformatics workflow-engine pipeline pipeline-framework nextflow cloud groovy sge slurm aws docker singularity hpc singularity-containers reproducible-science reproducible-research dataflow hello
Language:Groovy 3202
EpicGamesExt / BlenderTools
Blender addons that improve the game development workflow between Blender and Unreal.
python blender blender-addon unreal pipeline
Language:Python 3070
apptension / saas-boilerplate
SaaS Boilerplate - Open Source and free SaaS stack that lets you build SaaS products faster in React, Django and AWS. Focus on essential business logic instead of coding repeatable features!
aws aws-esc backend boilerplate business cdk django docker frontend hacktoberfest pipeline product python react saas serverless stack typescript vitejs workers
Language:TypeScript 2678
alibaba / pipcook
Machine learning platform for Web developers
js machine-learning pipeline tensorflow
Language:TypeScript 2581
EntilZha / PyFunctional
Python library for creating data pipelines with chain functional programming
python datascience pipeline data functional-programming
Language:Python 2479
InternLM / HuixiangDou
HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance
application assistant assistant-chat-bots chatbot dsl group-chat image-retrieval lark llm multimodal pipeline rag robot wechat
Language:Python 2441
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing
Language:Rust 2359
instill-core
instill-ai / instill-core
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
ai api cli developer-tools etl generative-ai golang gpt hacktoberfest llm low-code no-code open-source pipeline python stable-diffusion typescript unstructured-data
Language:Python 2294