akkikiki

nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

mamba

Mamba SSM architecture

llama-recipes

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization and Q&A. Supporting a number of candid inference solutions such as HF TGI, VLLM for local or cloud deployment. Demo apps to showcase Meta Llama3 for WhatsApp & Messenger.

mistral-inference

Official inference library for Mistral models

trl

Train transformer language models with reinforcement learning.

DiT

Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

alignment-handbook

Robust recipes to align language models with human and AI preferences

mergekit

Tools for merging pretrained large language models.

Vim

[ICML 2024] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

mamba-minimal

Simple, minimal implementation of the Mamba SSM in one file of PyTorch.

deep_learning_curriculum

Language model alignment-focused deep learning curriculum

nanotron

Minimalistic large language model 3D-parallelism training

shell-ai

LangChain powered shell command generator and runner CLI

Triton-Puzzles

Puzzles for learning Triton

mamba.py

A simple and efficient Mamba implementation in pure PyTorch and MLX.

template

This is the repository for the distill web framework

ultravox

recurrentgemma

Open weights language model from Google DeepMind, based on Griffin.

gpt_paper_assistant

GPT4 based personalized ArXiv paper assistant bot

open_lm

A repository for research on medium sized language models.

SAELens

Training Sparse Autoencoders on Language Models

CoLT5-attention

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

sae

Sparse autoencoders

ocr-post-correction

mixture-of-attention

Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts

train-with-fsdp

kotomamba

Mamba training library developed by kotoba technologies

parallelizing_linear_rnns

mamba-triton

MLLM-Judge

[ICML 2024 Oral] Official code repository for MLLM-as-a-Judge.