pipelines

There are 42 repositories under pipelines topic.

taipy
Avaiga / taipy
Turns Data and AI algorithms into production-ready web applications in no time.
automation data-engineering data-integration data-ops data-visualization datascience developer-tools hacktoberfest hacktoberfest2023 job-scheduler mlops orchestration pipeline pipelines python scenario scenario-analysis taipy-core taipy-gui workflow
Language:Python 18848
argoproj / argo-workflows
Workflow Engine for Kubernetes
workflow kubernetes argo dag knative airflow machine-learning argo-workflows workflow-engine hacktoberfest cloud-native cncf k8s gitops mlops batch-processing data-engineering pipelines
Language:Go 16165
mage-ai / mage-ai
🧙 Build, run, and manage data pipelines for integrating and transforming data.
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines orchestration data-integration sql spark dbt pipeline reverse-etl transformation
Language:Python 8517
concourse / concourse
Concourse is a container-based automation system written in Go. It's mostly used for CI/CD.
ci ci-cd concourse containerd containers continuous-delivery continuous-integration elm go pipelines runc
Language:Go 7653
zenml
zenml-io / zenml
ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.
agentops agents ai automl data-science deep-learning devops-tools genai llm llmops machine-learning metadata-tracking ml mlops pipelines production-ready pytorch tensorflow workflow zenml
Language:Python 4996
orchest / orchest
Build data pipelines, the easy way 🛠️
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker python data-pipelines orchest deployment kubernetes airflow dag etl etl-pipeline
Language:TypeScript 4143
polyaxon / polyaxon
MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
deep-learning machine-learning artificial-intelligence data-science reinforcement-learning kubernetes tensorflow pytorch keras mxnet caffe ml k8s jupyter notebook jupyterlab pipelines workflow mlops hyperparameter-optimization
3684
ploomber / ploomber
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
data-engineering data-science jupyter jupyter-notebooks machine-learning mlops notebooks papermill pipelines pycharm vscode workflow
Language:Python 3604
argoproj / argo-events
Event-driven Automation Framework for Kubernetes
kubernetes event-driven cloudevents workflows triggers automation-framework cloud-native argo pipelines event-source workflow-automation eventing-framework
Language:Go 2570
salesforce / TransmogrifAI
TransmogrifAI (pronounced trăns-mŏgˈrə-fī) is an AutoML library for building modular, reusable, strongly typed machine learning workflows on Apache Spark with minimal hand-tuning
ai automated-machine-learning automl dsl einstein estimators feature-engineering features machine-learning ml pipelines salesforce scala spark sparkml structured-data transformations transformers transmogrification transmogrify
Language:Scala 2269
meltano / meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk connectors integration tap taps target targets loaders data-pipelines singer data-engineering
Language:Python 2248
elyra-ai / elyra
Elyra extends JupyterLab with an AI centric approach.
notebooks notebook-jupyter jupyterlab jupyterlab-extensions ai machine-learning pipelines kubeflow-pipelines python elyra binder jupyterlab-notebooks kubeflow hacktoberfest jupyterlab-extension anaconda pypi docker apache-airflow airflow
Language:Python 1969
JuliaAI / MLJ.jl
A Julia machine learning framework
machine-learning julia pipelines tuning data-science tuning-parameters predictive-modeling classification regression statistics clustering stacking ensemble-learning pipeline
Language:Julia 1879
covalent
AgnostiqHQ / covalent
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
hpc workflow workflow-automation workflow-management quantum-computing hpc-applications python machine-learning pipelines covalent orchestration parallelization machinelearning machinelearning-python quantum quantum-machine-learning data-science data-pipeline deep-learning hacktoberfest
Language:Python 844
stacksimplify / azure-aks-kubernetes-masterclass
Azure AKS Kubernetes Masterclass
azure-devops aks azure-container-registry kubernetes pipelines
Language:HCL 796
natayadev / dataengineering-roadmap
Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨
ai big-data data data-engineering pipelines ingenieria-de-datos roadmap
786
substantic / rain
Framework for large distributed pipelines
pipelines distributed-computing python-api rust workflows
Language:Rust 753
onepanelio / onepanel
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
deeplearning pipelines machinelearning jupyterlab tensorflow pytorch tensorboard computer-vision mlops workflows aiops annotation hyperparameter-tuning inference training labeling ai etl
Language:Go 732
tenzir
tenzir / tenzir
Tenzir is the data pipeline engine for security teams.
incident-response threathunting siem soc security dataops investigation pcap secdataops netflow suricata zeek pipelines sigma hacktoberfest
Language:C++ 705
JulianKemmerer / PipelineC
A C-like hardware description language (HDL) adding high level synthesis(HLS)-like automatic pipelining as a language construct/compiler feature.
fpga hardware-description-language vhdl pipelines c python hardware hardware-description high-level-synthesis hls fpga-programming fpga-acceleration fpga-accelerators open-source-hardware
Language:VHDL 676
Azure / azure-devops-cli-extension
Azure DevOps Extension for Azure CLI
cicd agile git command-line command-line-tool azure devops azure-devops azure-devops-extension azure-cli azure-cli-extension boards pipelines repos artifacts
Language:Python 665
pypyr
pypyr / pypyr
pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.
pipeline pipeline-runner pipeline-processor pipelines-yaml devops task-runner script script-loader taskrunner automation ci cd ci-cd tool task-manager pipelines continuous-deployment continuous-integration python
Language:Python 637
mdubourg001 / glci
🦊 Test your Gitlab CI Pipelines changes locally using Docker.
gitlab ci gitlab-ci docker jobs pipelines
Language:JavaScript 580
tektoncd / triggers
Event triggering with Tekton!
tekton pipelines triggers hacktoberfest
Language:Go 572
live-keys / livekeys
Livekeys - Automation Platform with support for live scripting.
pipelines video-processing image-processing live-coding
Language:C++ 568
microsoft / psi
Platform for Situated Intelligence
multimodal streaming human-robot-interaction multimodal-interactions perception framework stream-processing component-library artificial-intelligence pipelines
Language:C# 562
acln0 / zerocopy
Zero-copy I/O primitives and pipelines for Go. Linux-specific.
splice tee linux go zero-copy pipelines
Language:Go 548
timoschwarzer / gitlab-monitor
A web-based monitor dashboard for GitLab CI
dashboard gitlab gitlabci hacktoberfest monitor pipelines
Language:Vue 495
pipefunc
pipefunc / pipefunc
Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪
pipeline-framework pipelines reproducible-research dag hpc parallel-computing slurm workflow-engine
Language:Python 427
harness / drone-cli
Command Line Tools for Drone CI
continuous-integration continuous-delivery pipelines
Language:Go 416
smooks / smooks
An extensible Java framework for building event-driven applications that break up XML and non-XML data into chunks for data integration
java smooks enterprise-integration etl sax pipelines event-driven xml stream-processing analytics chunking
Language:Java 412
thomhurst / ModularPipelines
Write your pipelines in C# !
build build-system build-tool build-tools csharp pipeline pipeline-deploy pipeline-framework pipelines dotnet dotnet-core dotnetcore
Language:C# 400
meta-pytorch / torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
pytorch machine-learning kubernetes slurm distributed-training pipelines components deep-learning python aws-batch ray airflow
Language:Python 399
nf-core / modules
Repository to host tool-specific module files for the Nextflow DSL2 community!
nf-core nextflow modules workflows pipelines dsl2 nf-test
Language:Nextflow 385
mkdkr
rosineygp / mkdkr
mkdkr = Makefile + Docker
makefile docker pipelines bash travis-ci cirleci github-actions jenkins gitlab-ci
Language:Shell 378
calebwin / pipelines
An experimental programming language for data flow
pipelines pipeline parallel language nim python compiler
Language:Nim 372