zhenyuhe00

ChunkLlama

[ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"

ShiArthur03

awesome-RLHF

A curated list of reinforcement learning with human feedback resources (continually updated)

Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

MEGABYTE-pytorch

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

GeoMFormer

Panacea

ml4se

A curated list of papers, theses, datasets, and tools related to the application of Machine Learning for Software Engineering

DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

infini-transformer-pytorch

Implementation of Infini-Transformer in Pytorch

ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

RULER

This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

Image_FMAP

SnapKV

mixture-of-depths

An unofficial implementation of "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"

DVMP

The official implementation of dual-view molecule pre-training.

O-GNN

llama3

The official Meta Llama 3 GitHub site

Transformer-M

[ICLR 2023] One Transformer Can Understand Both 2D & 3D Molecular Data (official implementation)

VAR

[GPT beats diffusion🔥] [scaling laws in visual generation📈] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

LESS

EasyKV

Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)

InfLLM

The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"

TOVA

Token Omission Via Attention

H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

fairseq2

FAIR Sequence Modeling Toolkit 2

EasyContext

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

BiPE

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation, ICML 2024

mergekit

Tools for merging pretrained large language models.