sara4dev

langchain

🦜🔗 Build context-aware reasoning applications

papers-we-love

Papers from the computer science community to read and discuss.

llama.cpp

LLM inference in C/C++

autogen

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

skywalking

APM, Application Performance Monitoring System

llm.c

LLM training in simple, raw C/CUDA

qdrant

Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

candle

Minimalist ML framework for Rust

triton

Development repository for the Triton language and compiler

ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models

mise

dev tools, env vars, task runner

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

nvtop

GPU & Accelerator process monitoring for AMD, Apple, Huawei, Intel, NVIDIA and Qualcomm

rancher-desktop

Container Management and Kubernetes on the Desktop

cutlass

CUDA Templates for Linear Algebra Subroutines

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

digger

Digger is an open source IaC orchestration tool. Digger allows you to run IaC in your existing CI pipeline ⚡️

bpftop

bpftop provides a dynamic real-time view of running eBPF programs. It displays the average runtime, events per second, and estimated total CPU % for each program.

lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

axlearn

An Extensible Deep Learning Library

inference

Reference implementations of MLPerf™ inference benchmarks

LearnRust

Rust Learning Resources

flashinfer

FlashInfer: Kernel Library for LLM Serving

optimum-nvidia

llmperf

LLMPerf is a library for validating and benchmarking LLMs

nxs-universal-chart

The Helm chart you can use to install any of your applications into Kubernetes/OpenShift

rules_oci

Bazel rules for building OCI containers

k8s-dra-driver

Dynamic Resource Allocation (DRA) for NVIDIA GPUs in Kubernetes