3d-parallelism

There are 0 repository under 3d-parallelism topic.

• xrsrke / pipegoose

  Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

  3d-parallelismdata-parallelismdistributed-optimizershuggingface-transformerslarge-scale-language-modelingmegatronmegatron-lmmixture-of-expertsmodel-parallelismmoepipeline-parallelismsequence-parallelismtensor-parallelismtransformerszero-1
  Language:Python 81