jinzex

microxcaling

PyTorch emulation library for Microscaling (MX)-compatible data formats

TensorRT-Model-Optimizer

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization and sparsity. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.

FP8

lightning-thunder

Make PyTorch models up to 40% faster! Thunder is a source to source compiler for PyTorch. It enables using different hardware executors at once; across one or thousands of GPUs.

unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

BitNet

Implementation of "BitNet: Scaling 1-bit Transformers for Large Language Models" in pytorch

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Megatron-LLaMA

Best practice for training LLaMA models in Megatron-LM

MatX

An efficient C++17 GPU numerical computing library with Python-like syntax

gemma_pytorch

The official PyTorch implementation of Google's Gemma models

gemma

Open weights LLM from Google DeepMind.

mlx

MLX: An array framework for Apple silicon

ieee754

Python module which finds the IEEE-754 representation of a floating point number.

MS-AMP

Microsoft Automatic Mixed Precision Library

float8_experimental

This repository contains the experimental PyTorch native float8 training UX

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

GLM-130B

GLM-130B: An Open Bilingual Pre-Trained Model (ICLR 2023)

DCGM

NVIDIA Data Center GPU Manager (DCGM) is a project for gathering telemetry and measuring the health of NVIDIA GPUs

cccl

CUDA Core Compute Libraries

tensorrtllm_backend

The Triton TensorRT-LLM Backend

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

llama

Inference code for Llama models

WeightWatcher

The WeightWatcher tool for predicting the accuracy of Deep Neural Networks

float16-simulator.js

A simulator for low-precision floating point calculations running in the browser

accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

flash-attention

Fast and memory-efficient exact attention

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

Megatron-LM

Ongoing research training transformer models at scale