eqy

nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

tinygrad

You like pytorch? You like micrograd? You love tinygrad! ❤️

pytorch_geometric

Graph Neural Network Library for PyTorch

discord.py

An API wrapper for Discord written in Python.

volkswagen

:see_no_evil: Volkswagen detects when your tests are being run in a CI server, and makes them pass.

RWKV-LM

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

FlexGen

Running large language models on a single GPU for throughput-oriented scenarios.

litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

nvtop

GPU & Accelerator process monitoring for AMD, Apple, Huawei, Intel, NVIDIA and Qualcomm

GPU-Puzzles

Solve puzzles. Learn CUDA.

min-dalle

min(DALL·E) is a fast, minimal port of DALL·E Mini to PyTorch

KataGo

GTP engine and self-play learning in Go

ng-video-lecture

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

Sophia

The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”

hidet

An open-source efficient deep learning framework/compiler, written in python.

nvbench

CUDA Kernel Benchmarking Library

Awesome-GPU

Awesome resources for GPUs

paxml

Pax is a Jax-based machine learning framework for training large scale models. Pax allows for advanced and fully configurable experimentation and parallelization, and has demonstrated industry leading model flop utilization rates.

Fuser

A Fusion Code Generator for NVIDIA GPUs (commonly known as "nvFuser")

zero-bubble-pipeline-parallelism

Zero Bubble Pipeline Parallelism

pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration

mntm

modular neural turing machines

reactive-bluefish-experiments

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in both training and inference.

OSDP-public

Composable + Tunable = Optimal

gen

graph generation and analysis stuff

pin

A Pin

dissertation

torchrec

Pytorch domain library for recommendation systems