gyeongchan-yun

Awesome_LLM_System-PaperList

Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inference acceleration, and related works will be gradually added in the future. Welcome contributions!

Megatron-Kwai

Ongoing research training transformer models at scale

AMP

(NeurIPS 2022) Automatically finding good model-parallel strategies, especially for complex models and clusters.

alpa

Training and serving large-scale neural networks with auto parallelization.

Megatron-LM

Artifact for DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines

python-patterns

A collection of design patterns/idioms in Python

awesome-AI-system

paper and its code for AI System

zero-bubble-pipeline-parallelism

Zero Bubble Pipeline Parallelism

optimizing-multitask-training-through-dynamic-pipelines

Official repository for the paper DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines

Optimus-CC

[ASPLOS'23] Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression

Hetu-Galvatron

Galvatron is an automatic distributed training system designed for Transformer models, including Large Language Models (LLMs). If you have any interests, please visit/star/fork https://github.com/PKU-DAIR/Hetu-Galvatron

FASOP

InternEvo

AcmeTrace

HexGen

Serving LLMs on heterogeneous decentralized clusters.

any-precision-llm

[ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

AdaQP

Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training

H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Parcae

ML-Papers-Explained

Explanation to key concepts in ML

ML-Papers-of-the-Week

🔥Highlighting the top ML papers every week.

llm-awq

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

hap

Temporal_Fusion_Transform

Pytorch Implementation of Google's TFT

tft-pytorch

Pytorch Temporal Fusion Transformers

tft-torch

A Python library that implements ״Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting״

QSync

Official resporitory for "IPDPS' 24 QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devices".

llm-papers

List of Large Lanugage Model Papers

Megatron-LLaMA

Best practice for training LLaMA models in Megatron-LM

artifacts