yuuxiaooqingg

ChunkLlama

[ICML'24] Data and code for our paper "Training-Free Long-Context Scaling of Large Language Models"

bowdpr

Codebase for [Paper] Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval

tensor_parallel

Automatically split your PyTorch models on multiple GPUs for training & inference

token_visualizer

Token level visualization tools for large language models

LongAlign

[EMNLP 2024] LongAlign: A Recipe for Long Context Alignment of LLMs

MiniCPM

MiniCPM3-4B: An edge-side LLM that surpasses GPT-3.5-Turbo.

st-moe-pytorch

Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch

TransnormerLLM

Official implementation of TransNormerLLM: A Faster and Better LLM

mutransformers

some common Huggingface transformers in maximal update parametrization (µP)

math-lm

CLEX

[ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models

MNBVC

MNBVC(Massive Never-ending BT Vast Chinese corpus)超大规模中文语料集。对标chatGPT训练的40T数据。MNBVC数据集不但包括主流文化，也包括各个小众文化甚至火星文的数据。MNBVC数据集包括新闻、作文、小说、书籍、杂志、论文、台词、帖子、wiki、古诗、歌词、商品介绍、笑话、糗事、聊天记录等一切形式的纯文本中文数据。