zhang-yingping

LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

ThunderKittens

Tile primitives for speedy kernels

tiny-universe

《大模型白盒子构建指南》：一个全手搓的Tiny-Universe

ccml

simple autodiff library

tiny-gpu

A minimal GPU design in Verilog to learn how GPUs work from the ground up

torchtitan

A native PyTorch Library for large model training

llm-reasoners

A library for advanced large language model reasoning

Shiro

📜 A minimalist personal website embodying the purity of paper and freshness of snow.

ipc

[Start here!] Flow-IPC - Modern C++ toolkit for high-speed inter-process communication (IPC)

llm.c

LLM training in simple, raw C/CUDA

ns3-ai

Enable the interaction between ns-3 and popular frameworks using Python, which mean you can train and test your AI algorithms in ns-3 without changing any frameworks you are using now!

Chinese-Resume-in-Typst

使用 Typst 编写的中文简历, 语法简洁, 样式美观, 开箱即用, 可选是否显示照片

fms-fsdp

Demonstrate throughput of PyTorch FSDP

makeMoE

From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

parallel-computing-tutorial

scattermoe

Triton-based implementation of Sparse Mixture of Experts.

how-to-optim-algorithm-in-cuda

how to optimize some algorithm in cuda.

LapisCV

📃 开箱即用的 Obsidian / Typora 简历

text-clustering

Easily embed, cluster and semantically label text datasets

LlamaGym

Fine-tune LLM agents with online reinforcement learning

LLMs-from-scratch

Implementing a ChatGPT-like LLM from scratch, step by step

hatchet

A distributed, fault-tolerant task queue

tutel

Tutel MoE: An Optimized Mixture-of-Experts Implementation

cookbook

Deep learning for dummies. All the practical details and useful utilities that go into working with real models.

NewPipe

A libre lightweight streaming front-end for Android.

so-large-lm

大模型基础: 一文了解大模型基础知识

glake

GLake: optimizing GPU memory management and IO transmission.

How_to_optimize_in_GPU

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

lectures

Material for cuda-mode lectures

nanotron

Minimalistic large language model 3D-parallelism training