BlackSamorez / tensor_parallel

Automatically split your PyTorch models on multiple GPUs for training & inference

BlackSamorez/tensor_parallel Issues

Compatibility with `transformers > 4.36`: error: `AttributeError: 'tuple' object has no attribute 'to_legacy_cache'`
Updated a month ago
Does tensor_parallel support multi-node tensor parallel training?
Updated 2 months ago5
Customized generate func support?
Updated 5 months ago
RuntimeError: NCCL Error 3: internal error
Updated 5 months ago1
Slow inference performance for large Llama models compared to naive MP
Updated 7 months ago26
tensor_parallel int4 LLM is not working since release v2.0.0
Updated 9 months ago
Now, does tensor_parallel no longer support the huggingface trainer?
Updated 9 months ago
Can I use tensor_parallel to inference for a GPTQ quantized model?
Updated a year ago
No implement of generate() when using models from hugging face.
Updated a year ago
TensorParallel object has no attribute save_pretrained
Updated a year ago
No output when using tensor_parallel
Updated a year ago1
How to use the model in a scenario where it is stored in the Safetenors format?
Closed a year ago
Out of GPU memory for two A10 GPUs
Closed a year ago1
AttributeError: object has no attribute 'devices'
Updated a year ago
ValueError: Model parameters were moved to incorrect devices, did call on model.cuda() or model.to(device)? If so, please avoid doing that
Updated a year ago
Max Recursion Error when using with lora
Updated a year ago2
Can I parallelize just one large layer?
Updated a year ago1
Segmentation fault (core dumped)
Updated a year ago
Support of 8-bit and 4-bit quantization
Closed a year ago1
Would it suitable for the multi-GPU parallel inference for llama2?
Updated a year ago
2x slowdown using TP
Updated a year ago
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Updated a year ago
distributed TP model forward output's requires_grad is False
Updated a year ago5
why raised cuda error?
Closed a year ago18
tensor_parallel method distributed=True
Updated a year ago2
model.generate() with inputs_embeds
Closed a year ago3
Error loading LLAMA model config
Closed a year ago
Issues if GPU > 2
Closed a year ago6
GPT-2 broken starting in v1.2.5
Closed a year ago1
When I try to do the tensor_parallel on NLLB from meta, there is an error:
Closed a year ago1
When I try to do the tensor_parallel on NLLB from meta, there is an error
Updated a year ago
Cloud Tensor_parallel add multiple accelerator inference support with torch.distributed?
Closed a year ago4
Example Question (got error) : Try new 40B LLMs demo in Kaggle
Closed a year ago2
Possibility to run on different GPUs
Closed a year ago2
Support for PEFT LoRA and 4-bit quantization
Closed a year ago6
Request to fix the content about parallelformers in README.
Closed a year ago1
Question on custom models
Updated a year ago23
Not work with 4bit quant
Closed a year ago6
TypeError when multi-thread inference using tensor_parallel
Closed a year ago1
Does tensor_parallel support the model inference concurrently or in multi-threads?
Closed a year ago2
Does tensor_parallel support data parallel and tensor parallel hybrid training?
Updated a year ago
Error in README.Md, hence not able to load model with limited memory.
Closed a year ago5
Torch version requirement
Closed a year ago4
Great work！ and can this work with deepspeedzero?
Updated a year ago
Huggingface Accelerate
Closed a year ago1
What is the difference between this project and autotp of deepspeed?
Closed a year ago1
cuda memory not evenly distributed between devices
Closed a year ago6
set distributed=True, return AttributeError: 'NoneType' object
Closed a year ago2
How to load lora weights？
Closed a year ago13
Support LLaMA Models, including HuggingFace-adapted variants
Closed a year ago7