pipelines

There are 33 repositories under pipelines topic.

argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 14373
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 8824
concourse / concourse
Concourse is a container-based continuous thing-doer written in Go.
ci ci-cd concourse continuous-delivery continuous-integration elm go hacktoberfest pipelines
Language:Go 7203
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
artificial-intelligence data data-engineering data-integration data-pipelines data-science dbt elt etl machine-learning orchestration pipeline pipelines python reverse-etl spark sql transformation
Language:Python 7149
orchest / orchest
Build data pipelines, the easy way 🛠️
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker python data-pipelines orchest deployment kubernetes airflow dag etl etl-pipeline
Language:TypeScript 4021
zenml
zenml-io / zenml
ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.
mlops machine-learning data-science production-ready devops-tools zenml pipelines metadata-tracking deep-learning pytorch tensorflow ml ai automl hacktoberfest workflow llm llmops
Language:Python 3686
polyaxon / polyaxon
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
deep-learning machine-learning artificial-intelligence data-science reinforcement-learning kubernetes tensorflow pytorch keras mxnet caffe ml k8s jupyter notebook jupyterlab pipelines workflow mlops hyperparameter-optimization
3493
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode pycharm notebooks
Language:Python 3390
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
argo automation-framework cloud-native cloudevents event-driven event-source eventing-framework kubernetes pipelines triggers workflow-automation workflows
Language:Go 2250
salesforce / TransmogrifAI
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
ml automl transformations estimators dsl pipelines machine-learning scala salesforce einstein features feature-engineering spark sparkml ai automated-machine-learning transmogrification transmogrify structured-data transformers
Language:Scala 2224
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
notebooks notebook-jupyter jupyterlab jupyterlab-extensions ai machine-learning pipelines kubeflow-pipelines python elyra binder jupyterlab-notebooks kubeflow hacktoberfest jupyterlab-extension anaconda pypi docker apache-airflow airflow
Language:Python 1780
JuliaAI / MLJ.jl
A Julia machine learning framework
machine-learning julia pipelines tuning data-science tuning-parameters predictive-modeling classification regression statistics clustering stacking ensemble-learning pipeline
Language:Julia 1727
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk connectors integration tap taps target targets loaders data-pipelines singer data-engineering
Language:Python 1611
substantic / rain
Framework for large distributed pipelines
distributed-computing pipelines python-api rust workflows
Language:Rust 738
onepanelio / onepanel
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
deeplearning pipelines machinelearning jupyterlab tensorflow pytorch tensorboard computer-vision mlops workflows aiops annotation hyperparameter-tuning inference training labeling ai etl
Language:Go 705
covalent
AgnostiqHQ / covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent orchestration parallelization machinelearning machinelearning-python quantum quantum-machine-learning data-science data-pipeline deep-learning hacktoberfest
Language:Python 702
tenzir
tenzir / tenzir
Open source security data pipelines.
incident-response threathunting siem soc security dataops investigation pcap secdataops netflow suricata zeek pipelines sigma
Language:C++ 615
Azure / azure-devops-cli-extension
Azure DevOps Extension for Azure CLI
cicd agile git command-line command-line-tool azure devops azure-devops azure-devops-extension azure-cli azure-cli-extension boards pipelines repos artifacts
Language:Python 611
stacksimplify / azure-aks-kubernetes-masterclass
Azure AKS Kubernetes Masterclass
azure-devops aks azure-container-registry kubernetes pipelines
Language:HCL 607
pypyr
pypyr / pypyr
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
pipeline pipeline-runner pipeline-processor pipelines-yaml devops task-runner script script-loader taskrunner automation ci cd ci-cd tool task-manager pipelines continuous-deployment continuous-integration python
Language:Python 571
mdubourg001 / glci
🦊 Test your Gitlab CI Pipelines changes locally using Docker.
gitlab ci gitlab-ci docker jobs pipelines
Language:JavaScript 566
live-keys / livekeys
Livekeys - Automation Platform with support for live scripting.
pipelines video-processing image-processing live-coding
Language:C++ 562
JulianKemmerer / PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
fpga hardware-description-language vhdl pipelines c python hardware hardware-description high-level-synthesis hls fpga-programming fpga-acceleration fpga-accelerators open-source-hardware
Language:Python 548
tektoncd / triggers
Event triggering with Tekton!
hacktoberfest pipelines tekton triggers
Language:Go 537
acln0 / zerocopy
Zero-copy I/O primitives and pipelines for Go. Linux-specific.
splice tee linux go zero-copy pipelines
Language:Go 527
microsoft / psi
Platform for Situated Intelligence
multimodal streaming human-robot-interaction multimodal-interactions perception framework stream-processing component-library artificial-intelligence pipelines
Language:C# 519
timoschwarzer / gitlab-monitor
A web-based monitor dashboard for GitLab CI
gitlab dashboard monitor gitlabci pipelines hacktoberfest
Language:Vue 474
natayadev / dataengineering-roadmap
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
ai big-data data data-engineering ingenieria-de-datos pipelines roadmap
419
harness / drone-cli
Command Line Tools for Drone CI
continuous-integration continuous-delivery pipelines
Language:Go 401
smooks / smooks
Extensible data integration Java framework for building XML and non-XML fragment-based applications
analytics big-data enterprise-integration etl event-driven java pipelines sax smooks stream-processing xml
Language:Java 385
calebwin / pipelines
An experimental programming language for data flow
compiler language nim parallel pipeline pipelines python
Language:Nim 373
mkdkr
rosineygp / mkdkr
mkdkr = Makefile + Docker
makefile docker pipelines bash travis-ci cirleci github-actions jenkins gitlab-ci
Language:Shell 361
zorbash / opus
A framework for pluggable business logic components
dsl elixir pipelines railway-oriented-programming
Language:Elixir 351
dmmiller612 / sparktorch
Train and run Pytorch models on Apache Spark.
pytorch sparktorch deep-learning apache-spark pipelines inference distributed-computing
Language:Python 336
pytorch / torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
airflow aws-batch components deep-learning distributed-training kubernetes machine-learning pipelines python pytorch ray slurm
Language:Python 303
cuebook / cuelake
Use SQL to build ELT pipelines on a data lakehouse.
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion apache-spark spark-sql upsert incremental-updates data-transfer pipelines data-pipeline zeppelin-notebook sql
Language:JavaScript 283