GHGmc2

llama3

The official Meta Llama 3 GitHub site

llm.c

LLM training in simple, raw C/CUDA

Awesome-LLM

Awesome-LLM: a curated list of Large Language Model

llama3-from-scratch

llama3 implementation one matrix multiplication at a time

accelerate

🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

tiny-gpu

A minimal GPU design in Verilog to learn how GPUs work from the ground up

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

optax

Optax is a gradient processing and optimization library for JAX.

fastmoe

A fast MoE impl for PyTorch

ThunderKittens

Tile primitives for speedy kernels

maxtext

A simple, performant and scalable Jax LLM!

torchtitan

A native PyTorch Library for large model training

Skywork

Skywork series models are pre-trained on 3.2TB of high-quality multilingual (mainly Chinese and English) and code data. We have open-sourced the model, training data, evaluation data, evaluation methods, etc. 天工系列模型在3.2TB高质量多语言和代码数据上进行预训练。我们开源了模型参数，训练数据，评估数据，评估方法。

nanotron

Minimalistic large language model 3D-parallelism training

EAGLE

Official Implementation of EAGLE-1 and EAGLE-2

LongLM

[ICML'24 Spotlight] LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

megalodon

Reference implementation of Megalodon 7B model

ByteTransformer

optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052

InternEvo

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

Awesome-DL-Scheduling-Papers

General-World-Models-Survey

FlagGems

FlagGems is an operator library for large language models implemented in Triton Language.

LightSeq

Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers

nccl-rdma-sharp-plugins

RDMA and SHARP plugins for nccl library

modern-latex

A short guide to LaTeX that avoids legacy cruft.

ml-systems-papers

Curated collection of papers in machine learning systems

grouped_gemm

PyTorch bindings for CUTLASS grouped GEMM.

occl

Parcae