tensor-parallelism

There are 0 repository under tensor-parallelism topic.

bigscience-workshop / petals
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
bloom chatbot deep-learning distributed-systems falcon gpt guanaco language-models large-language-models llama machine-learning mixtral neural-networks nlp pipeline-parallelism pretrained-models pytorch tensor-parallelism transformer volunteer-computing
Language:Python 9316
InternLM / InternEvo
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
910b deepspeed-ulysses flash-attention gemma internlm internlm2 llama3 llava llm-framework llm-training multi-modal pipeline-parallelism pytorch ring-attention sequence-parallelism tensor-parallelism transformers-models zero3
Language:Python 321
kaiyuyue / torchshard
Slicing a PyTorch Tensor Into Parallel Shards
pytorch model-parallelism tensor-parallelism
Language:Python 296
xrsrke / pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
3d-parallelism data-parallelism distributed-optimizers huggingface-transformers large-scale-language-modeling megatron megatron-lm mixture-of-experts model-parallelism moe pipeline-parallelism sequence-parallelism tensor-parallelism transformers zero-1
Language:Python 81
aniquetahir / JORA
JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
jax large-language-models lora machine-learning tensor-parallelism
Language:Python 32
fattorib / transformer_shmap
Tensor Parallelism with JAX + Shard Map
gpt jax pjit tensor-parallelism tpu transformers shmap
Language:Python 11
ShinoharaHare / LLM-Training
A distributed training framework for large language models powered by Lightning.
distributed-training fine-tuning instruction-tuning llama llm llm-training phi pre-training tensor-parallelism transformer llm-finetuning
Language:Python 11
AlibabaPAI / FlashModels
Fast and easy distributed model training examples.
distributed-training xla data-parallelism deep-learning fsdp model-parallelism pytorch sequence-parallelism tensor-parallelism zero llm
Language:Python 10
InternLM / InternEvo-HFModels
Democratizing huggingface model training with InternEvo
huggingface model-parallelism sequence-parallelism tensor-parallelism zero3 llm
Language:Python 6
NiuHuangxiaozi / Deep-Learning-Parallelism
This repository outlines a comprehensive guide for training a distributed deep learning model.
allreduce ddp deepspeed pipline ps pytorch tensor-parallelism
Language:Python 1