ifromeast / LLMTrainer

A comparison of pretraining framework for LLM

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LLMTrainer

对比 LLM 的多种预训练方案的性能,方案包括:

  • Torch DDP
  • FSDP
  • DeepSpeed
  • FlashAttention V2
  • ColossalAI
  • Sophia Optimizer
  • Lightning Fabric

详细介绍见 https://zhuanlan.zhihu.com/p/648094197?

About

A comparison of pretraining framework for LLM


Languages

Language:Python 87.6%Language:Shell 12.4%