pipelines

There are 31 repositories under pipelines topic.

argoproj / argo-workflows
Workflow Engine for Kubernetes
airflow argo argo-workflows batch-processing cloud-native cncf dag data-engineering gitops hacktoberfest k8s knative kubernetes machine-learning mlops pipelines workflow workflow-engine
Language:Go 14266
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines python taipy-gui workflow taipy-core hacktoberfest hacktoberfest2023 data-integration job-scheduler scenario scenario-analysis
Language:Python 8315
concourse / concourse
Concourse is a container-based continuous thing-doer written in Go.
go ci pipelines elm concourse continuous-integration continuous-delivery ci-cd hacktoberfest
Language:Go 7166
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 6977
orchest / orchest
Build data pipelines, the easy way 🛠️
airflow cloud dag data-pipelines data-science deployment docker etl etl-pipeline ide jupyter jupyterlab kubernetes machine-learning notebooks orchest pipelines python self-hosted
Language:TypeScript 4021
zenml
zenml-io / zenml
ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.
mlops machine-learning data-science production-ready devops-tools zenml pipelines metadata-tracking deep-learning pytorch tensorflow ml ai automl hacktoberfest workflow llm llmops
Language:Python 3647
polyaxon / polyaxon
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
deep-learning machine-learning artificial-intelligence data-science reinforcement-learning kubernetes tensorflow pytorch keras mxnet caffe ml k8s jupyter notebook jupyterlab pipelines workflow mlops hyperparameter-optimization
3479
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow
Language:Python 3369
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
kubernetes event-driven cloudevents workflows triggers automation-framework cloud-native argo pipelines event-source workflow-automation eventing-framework
Language:Go 2230
salesforce / TransmogrifAI
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
ai automated-machine-learning automl dsl einstein estimators feature-engineering features machine-learning ml pipelines salesforce scala spark sparkml structured-data transformations transformers transmogrification transmogrify
Language:Scala 2227
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
notebooks notebook-jupyter jupyterlab jupyterlab-extensions ai machine-learning pipelines kubeflow-pipelines python elyra binder jupyterlab-notebooks kubeflow hacktoberfest jupyterlab-extension anaconda pypi docker apache-airflow airflow
Language:Python 1770
alan-turing-institute / MLJ.jl
A Julia machine learning framework
machine-learning julia pipelines tuning data-science tuning-parameters predictive-modeling classification regression statistics clustering stacking ensemble-learning pipeline
Language:Julia 1718
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk connectors integration tap taps target targets loaders data-pipelines singer data-engineering
Language:Python 1581
substantic / rain
Framework for large distributed pipelines
pipelines distributed-computing python-api rust workflows
Language:Rust 737
onepanelio / onepanel
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
deeplearning pipelines machinelearning jupyterlab tensorflow pytorch tensorboard computer-vision mlops workflows aiops annotation hyperparameter-tuning inference training labeling ai etl
Language:Go 704
covalent
AgnostiqHQ / covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent orchestration parallelization machinelearning machinelearning-python quantum quantum-machine-learning data-science data-pipeline deep-learning hacktoberfest
Language:Python 687
Azure / azure-devops-cli-extension
Azure DevOps Extension for Azure CLI
cicd agile git command-line command-line-tool azure devops azure-devops azure-devops-extension azure-cli azure-cli-extension boards pipelines repos artifacts
Language:Python 609
tenzir
tenzir / tenzir
Open source security data pipelines.
incident-response threathunting siem soc security dataops investigation pcap secdataops netflow suricata zeek pipelines sigma
Language:C++ 609
stacksimplify / azure-aks-kubernetes-masterclass
Azure AKS Kubernetes Masterclass
azure-devops aks azure-container-registry kubernetes pipelines
Language:HCL 600
pypyr
pypyr / pypyr
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
pipeline pipeline-runner pipeline-processor pipelines-yaml devops task-runner script script-loader taskrunner automation ci cd ci-cd tool task-manager pipelines continuous-deployment continuous-integration python
Language:Python 568
mdubourg001 / glci
🦊 Test your Gitlab CI Pipelines changes locally using Docker.
ci docker gitlab gitlab-ci jobs pipelines
Language:JavaScript 567
live-keys / livekeys
Livekeys - Automation Platform with support for live scripting.
pipelines video-processing image-processing live-coding
Language:C++ 561
JulianKemmerer / PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
c fpga fpga-acceleration fpga-accelerators fpga-programming hardware hardware-description hardware-description-language high-level-synthesis hls open-source-hardware pipelines python vhdl
Language:Python 541
tektoncd / triggers
Event triggering with Tekton!
tekton pipelines triggers hacktoberfest
Language:Go 532
acln0 / zerocopy
Zero-copy I/O primitives and pipelines for Go. Linux-specific.
splice tee linux go zero-copy pipelines
Language:Go 523
microsoft / psi
Platform for Situated Intelligence
artificial-intelligence component-library framework human-robot-interaction multimodal multimodal-interactions perception pipelines stream-processing streaming
Language:C# 507
timoschwarzer / gitlab-monitor
A web-based monitor dashboard for GitLab CI
dashboard gitlab gitlabci hacktoberfest monitor pipelines
Language:Vue 473
natayadev / dataengineering-roadmap
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
ai big-data data data-engineering ingenieria-de-datos pipelines roadmap
408
harness / drone-cli
Command Line Tools for Drone CI
continuous-integration continuous-delivery pipelines
Language:Go 398
smooks / smooks
Extensible data integration Java framework for building XML and non-XML fragment-based applications
java smooks enterprise-integration big-data etl sax pipelines event-driven xml stream-processing analytics
Language:Java 383
calebwin / pipelines
An experimental programming language for data flow
pipelines pipeline parallel language nim python compiler
Language:Nim 372
mkdkr
rosineygp / mkdkr
mkdkr = Makefile + Docker
makefile docker pipelines bash travis-ci cirleci github-actions jenkins gitlab-ci
Language:Shell 359
zorbash / opus
A framework for pluggable business logic components
railway-oriented-programming pipelines dsl elixir
Language:Elixir 348
dmmiller612 / sparktorch
Train and run Pytorch models on Apache Spark.
pytorch sparktorch deep-learning apache-spark pipelines inference distributed-computing
Language:Python 334
pytorch / torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
airflow aws-batch components deep-learning distributed-training kubernetes machine-learning pipelines python pytorch ray slurm
Language:Python 295
cuebook / cuelake
Use SQL to build ELT pipelines on a data lakehouse.
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion apache-spark spark-sql upsert incremental-updates data-transfer pipelines data-pipeline zeppelin-notebook sql
Language:JavaScript 283