pipelines

There are 34 repositories under pipelines topic.

argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 14619
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 10009
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 7490
concourse / concourse
Concourse is a container-based continuous thing-doer written in Go.
ci ci-cd concourse continuous-delivery continuous-integration elm go hacktoberfest pipelines
Language:Go 7262
orchest / orchest
Build data pipelines, the easy way 🛠️
airflow cloud dag data-pipelines data-science deployment docker etl etl-pipeline ide jupyter jupyterlab kubernetes machine-learning notebooks orchest pipelines python self-hosted
Language:TypeScript 4035
zenml
zenml-io / zenml
ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.
mlops machine-learning data-science production-ready devops-tools zenml pipelines metadata-tracking deep-learning pytorch tensorflow ml ai automl hacktoberfest workflow llm llmops
Language:Python 3834
polyaxon / polyaxon
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
deep-learning machine-learning artificial-intelligence data-science reinforcement-learning kubernetes tensorflow pytorch keras mxnet caffe ml k8s jupyter notebook jupyterlab pipelines workflow mlops hyperparameter-optimization
3520
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow
Language:Python 3442
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
kubernetes event-driven cloudevents workflows triggers automation-framework cloud-native argo pipelines event-source workflow-automation eventing-framework
Language:Go 2291
salesforce / TransmogrifAI
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
ml automl transformations estimators dsl pipelines machine-learning scala salesforce einstein features feature-engineering spark sparkml ai automated-machine-learning transmogrification transmogrify structured-data transformers
Language:Scala 2228
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
ai airflow anaconda apache-airflow binder docker elyra hacktoberfest jupyterlab jupyterlab-extension jupyterlab-extensions jupyterlab-notebooks kubeflow kubeflow-pipelines machine-learning notebook-jupyter notebooks pipelines pypi python
Language:Python 1807
JuliaAI / MLJ.jl
A Julia machine learning framework
machine-learning julia pipelines tuning data-science tuning-parameters predictive-modeling classification regression statistics clustering stacking ensemble-learning pipeline
Language:Julia 1755
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk connectors integration tap taps target targets loaders data-pipelines singer data-engineering
Language:Python 1709
substantic / rain
Framework for large distributed pipelines
pipelines distributed-computing python-api rust workflows
Language:Rust 741
covalent
AgnostiqHQ / covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent orchestration parallelization machinelearning machinelearning-python quantum quantum-machine-learning data-science data-pipeline deep-learning hacktoberfest
Language:Python 732
onepanelio / onepanel
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
deeplearning pipelines machinelearning jupyterlab tensorflow pytorch tensorboard computer-vision mlops workflows aiops annotation hyperparameter-tuning inference training labeling ai etl
Language:Go 709
stacksimplify / azure-aks-kubernetes-masterclass
Azure AKS Kubernetes Masterclass
aks azure-container-registry azure-devops kubernetes pipelines
Language:HCL 634
tenzir
tenzir / tenzir
Open source security data pipelines.
incident-response threathunting siem soc security dataops investigation pcap secdataops netflow suricata zeek pipelines sigma
Language:C++ 623
Azure / azure-devops-cli-extension
Azure DevOps Extension for Azure CLI
cicd agile git command-line command-line-tool azure devops azure-devops azure-devops-extension azure-cli azure-cli-extension boards pipelines repos artifacts
Language:Python 618
pypyr
pypyr / pypyr
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
pipeline pipeline-runner pipeline-processor pipelines-yaml devops task-runner script script-loader taskrunner automation ci cd ci-cd tool task-manager pipelines continuous-deployment continuous-integration python
Language:Python 578
JulianKemmerer / PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
fpga hardware-description-language vhdl pipelines c python hardware hardware-description high-level-synthesis hls fpga-programming fpga-acceleration fpga-accelerators open-source-hardware
Language:Python 570
mdubourg001 / glci
🦊 Test your Gitlab CI Pipelines changes locally using Docker.
gitlab ci gitlab-ci docker jobs pipelines
Language:JavaScript 568
live-keys / livekeys
Livekeys - Automation Platform with support for live scripting.
pipelines video-processing image-processing live-coding
Language:C++ 563
tektoncd / triggers
Event triggering with Tekton!
tekton pipelines triggers hacktoberfest
Language:Go 546
acln0 / zerocopy
Zero-copy I/O primitives and pipelines for Go. Linux-specific.
splice tee linux go zero-copy pipelines
Language:Go 536
microsoft / psi
Platform for Situated Intelligence
multimodal streaming human-robot-interaction multimodal-interactions perception framework stream-processing component-library artificial-intelligence pipelines
Language:C# 530
timoschwarzer / gitlab-monitor
A web-based monitor dashboard for GitLab CI
gitlab dashboard monitor gitlabci pipelines hacktoberfest
Language:Vue 478
natayadev / dataengineering-roadmap
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
ai big-data data data-engineering pipelines ingenieria-de-datos roadmap
460
harness / drone-cli
Command Line Tools for Drone CI
continuous-integration continuous-delivery pipelines
Language:Go 403
smooks / smooks
Extensible data integration Java framework for building XML and non-XML fragment-based applications
analytics big-data enterprise-integration etl event-driven java pipelines sax smooks stream-processing xml
Language:Java 392
calebwin / pipelines
An experimental programming language for data flow
pipelines pipeline parallel language nim python compiler
Language:Nim 372
mkdkr
rosineygp / mkdkr
mkdkr = Makefile + Docker
makefile docker pipelines bash travis-ci cirleci github-actions jenkins gitlab-ci
Language:Shell 364
zorbash / opus
A framework for pluggable business logic components
railway-oriented-programming pipelines dsl elixir
Language:Elixir 354
dmmiller612 / sparktorch
Train and run Pytorch models on Apache Spark.
pytorch sparktorch deep-learning apache-spark pipelines inference distributed-computing
Language:Python 337
pytorch / torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
pytorch machine-learning kubernetes slurm distributed-training pipelines components deep-learning python aws-batch ray airflow
Language:Python 311
sharpliner / sharpliner
Use C# instead of YAML to define your Azure DevOps pipelines
azure-devops-pipelines pipelines yaml azure-devops csharp dotnet
Language:C# 287