megatron-lm

There are 0 repository under megatron-lm topic.

alibaba / Megatron-LLaMA
Best practice for training LLaMA models in Megatron-LM
deepspeed distributed-training llama llm megatron-lm pretraining pytorch
Language:Python 543
shreyansh26 / Annotated-ML-Papers
Annotations of the interesting ML papers I read
deep-learning machine-learning research-paper annotated-paper bert nlp gpt transformers xlnet gpt-2 megatron-lm papers-annotations
188
xrsrke / pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
megatron megatron-lm transformers 3d-parallelism data-parallelism pipeline-parallelism tensor-parallelism model-parallelism zero-1 large-scale-language-modeling huggingface-transformers distributed-optimizers mixture-of-experts moe sequence-parallelism
Language:Python 73
MoFHeka / LLaMA-Megatron
A LLaMA1/LLaMA12 Megatron implement.
llama llama2 llm llm-training megatron megatron-lm pytorch
Language:Python 24
Beomi / megatronlm_dataset_autotokenizer
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
megatron-lm gpt-neox tokenizers transformers
Language:Python 5
GoogleCloudPlatform / nvidia-nemo-on-gke
Training NVIDIA NeMo Megatron Large Language Model (LLM) using NeMo Framework on Google Kubernetes Engine
gke megatron-lm nvidia nvidia-gpu nvidia-nemo
Language:HCL 4
GJ98 / Megatron-LM
Megatron-LM implemented by PyTorch
megatron-lm nlp pytorch
Language:Python 1