pipeline

There are 84 repositories under pipeline topic.

jina
jina-ai / jina
☁️ Build multimodal AI applications with cloud-native stack
cloud-native cncf deep-learning docker fastapi framework generative-ai grpc jaeger kubernetes llmops machine-learning microservice mlops multimodal neural-search opentelemetry orchestration pipeline prometheus
Language:Python 20135
vector
vectordotdev / vector
A high-performance observability data pipeline.
events forwarder logs metrics observability parser pipeline router rust stream-processing vector
Language:Rust 16639
argoproj / argo-cd
Declarative Continuous Deployment for Kubernetes
argo kubernetes continuous-deployment gitops continuous-delivery docker cd cicd pipeline devops ci-cd argo-cd ksonnet helm hacktoberfest
Language:Go 16291
PrefectHQ / prefect
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
automation data data-engineering data-ops data-science infrastructure ml-ops observability orchestration pipeline prefect python workflow workflow-engine
Language:Python 14795
airbytehq / airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration elt bigquery redshift snowflake data-pipeline mssql mysql postgresql s3 self-hosted
Language:Python 14241
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 9517
prql
PRQL / prql
PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement
data pipeline sql
Language:Rust 9470
kedro
kedro-org / kedro
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
pipeline kedro hacktoberfest mlops experiment-tracking python machine-learning machine-learning-engineering
Language:Python 9383
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 8814
tektoncd / pipeline
A cloud-native Pipeline resource.
cdf hacktoberfest kubernetes pipeline tekton
Language:Go 8299
proposal-pipeline-operator
tc39 / proposal-pipeline-operator
A proposal for adding a useful pipe operator to JavaScript.
javascript tc39 proposal pipeline operator syntax
Language:HTML 7390
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 7131
projectdiscovery / httpx
httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library.
bugbounty cli cybersecurity hacktoberfest http lib osint pentest-tool pipeline ssl-certificate
Language:Go 6914
brunch / brunch
:fork_and_knife: Web applications made easy. Since 2011.
brunch build-automation javascript pipeline workflow
Language:JavaScript 6814
kestra
kestra-io / kestra
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
data data-engineering data-integration data-orchestration data-orchestrator data-pipeline data-quality elt etl low-code orchestration pipeline reverse-etl scheduler workflow workflow-engine
Language:Java 6588
nteract / papermill
📚 Parameterize, execute, and analyze notebooks
jupyter notebooks notebook-generator nteract publishing pipeline notebook python r julia scala
Language:Python 5650
gaia-pipeline / gaia
Build powerful pipelines in any programming language.
go pipeline automation java python cplusplus build deployment kubernetes continuous-integration continuous-delivery continuous-testing devops devops-tools ruby distributed-systems gitops
Language:Go 5167
iam-veeramalla / Jenkins-Zero-To-Hero
Install Jenkins, configure Docker as slave, set up cicd, deploy applications to k8s using Argo CD in GitOps way.
argocd cicd docker gitops jenkins kubernetes pipeline
Language:Python 4984
jx
jenkins-x / jx
Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
accelerator cicd continuous-delivery continuous-integration devops environments gitops golang hacktoberfest kubernetes openshift pipeline tekton
Language:Go 4522
GameDevMind
gonglei007 / GameDevMind
最全面的游戏开发技术图谱。帮助游戏开发者们在已知问题上节省时间，省出更多的精力投入到更有创造性的工作中去。
game-development unity3d 3d programming unreal-engine unrealengine game shader game-server framework mmorpg roadmap scrum pipeline game-framework cpp management devops gamedev unity
Language:Shell 4463
Production-Level-Deep-Learning
alirezadir / Production-Level-Deep-Learning
A guideline for building practical production-level deep learning systems to be deployed in real world applications.
ai artificial-intelligence deep-learning deployment kubeflow machine-learning pipeline practical-machine-learning production-system scalable-applications system-design tfx
4214
marimo-team / marimo
A reactive notebook for Python — run reproducible experiments, execute as a script, deploy as an app, and version with git.
artificial-intelligence dag data-science data-visualization dataflow developer-tools machine-learning notebooks pipeline python reactive web-app
Language:Python 4203
kubeflow / pipelines
Machine Learning Pipelines for Kubeflow
data-science kubeflow kubeflow-pipelines kubernetes machine-learning mlops pipeline
Language:Python 3455
towhee-io / towhee
Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.
machine-learning convolutional-networks embedding-vectors embeddings computer-vision image-processing video-processing feature-extraction image-retrieval unstructured-data feature-vector transformer milvus towhee vision-transformer vit pipeline llm
Language:Python 3013
EpicGamesExt / BlenderTools
Blender addons that improve the game development workflow between Blender and Unreal.
blender blender-addon pipeline python unreal
Language:Python 2617
cube-studio
tencentmusic / cube-studio
cube studio开源云原生一站式机器学习/深度学习AI平台，支持sso登录，多租户/多项目组，大数据平台对接，notebook在线开发，拖拉拽任务流pipeline编排，多机多卡分布式训练，超参搜索，推理服务VGPU，边缘计算，serverless，标注平台，自动化标注，数据集管理，大模型微调，vllm大模型推理，llmops，私有知识库，AI模型应用商店，支持模型一键开发/推理/微调，支持国产cpu/gpu/npu芯片，支持RDMA，支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/spark/ray/volcano分布式
kubernetes inference mlops workflow ai pytorch spark argo kubeflow automl aihub gpt llmops notebook pipeline vgpu
Language:Jupyter Notebook 2561
nextflow
nextflow-io / nextflow
A DSL for data-driven computational pipelines
bioinformatics workflow-engine pipeline pipeline-framework nextflow cloud groovy sge slurm aws docker singularity hpc singularity-containers reproducible-science reproducible-research dataflow hello
Language:Groovy 2555
alibaba / pipcook
Machine learning platform for Web developers
js machine-learning pipeline tensorflow
Language:TypeScript 2506
yuanzhoulvpi2017 / zero_nlp
中文nlp解决方案(大模型、数据、模型、训练、推理)
bert nlp transformers gpt2 chatglm-6b clip gpt gpt-neox pytorch text-generation dolly bloom huggingface-transformers falcon llama2 pipeline
Language:Python 2506
EntilZha / PyFunctional
Python library for creating data pipelines with chain functional programming
python datascience pipeline data functional-programming
Language:Python 2341
mara / mara-pipelines
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
data data-integration etl pipeline postgresql python
Language:Python 2053
firecow / gitlab-ci-local
Tired of pushing to test your .gitlab-ci.yml?
gitlab local pipeline git ci cd push untracked uncomitted gitlab-ci
Language:TypeScript 1898
instill-core
instill-ai / instill-core
🔮 Instill Core is an open-source no-/low-code data, model, and pipeline orchestration platform
ai api cli developer-tools etl generative-ai golang gpt hacktoberfest llm low-code no-code open-source pipeline python stable-diffusion typescript unstructured-data
Language:Makefile 1895
hu17889 / go_spider
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
spider crawler go schedule pipeline
Language:Go 1829
edyoda / data-science-complete-tutorial
For extensive instructor led learning
machine-learning scikit-learn linear-regression nearest-neighbors feature-selection decision-trees numpy pandas pipeline
Language:Jupyter Notebook 1775
go-streams
reugn / go-streams
A lightweight stream processing library for Go
aerospike data-pipeline data-stream etl kafka kafka-streams low-code nats-streaming pipeline pulsar redis stream-processing stream-processor streaming-api streaming-data streams throttling websocket windowing workflow
Language:Go 1761