HuangLK / transpeeder

train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism

HuangLK/transpeeder Issues

Running 7b succeed. next 30B
Updated 8 months ago10
拆分成4个分片模型，如何设置每个GPU只加载一个分片
Updated 8 months ago
size mismatch
Updated 10 months ago1
ImportError: cannot import name 'flash_attn_unpadded_qkvpacked_func' from 'flash_attn.flash_attn_interface'
Closed a year ago3
error when use zero1
Updated a year ago1
how can run it with 24G GPU card like 3090
Updated a year ago2
Flash attention integration failed
Updated a year ago
How to set distributed sampler when using hybrid training of pipeline parallelism and data parallel
Closed a year ago2
pipeline model的一些问题
Closed a year ago2
RuntimeError: element 1 of tensors does not require grad and does not have a grad_fn
Closed a year ago
support bf16？
Updated a year ago4
flash_attn_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol
Closed a year ago
Why do we need to add 1 to the vocab_size when constructing the model?
Updated a year ago2
Output is not getting saved
Updated a year ago1
File not found error
Closed a year ago3
attention mask
Closed a year ago5
请问现在还不支持张量并行么？只支持流水线并行和数据并行？
Updated a year ago2
请问有计划支持peft吗
Updated a year ago1
hidden_states=bool变量
Updated a year ago1
train.py中加载checkpoint似乎没效
Closed a year ago
四卡训7B-llama清空缓存再训练报错
Closed a year ago1
使用4块3090全量微调7B-llama时发现启动训练阶段非常缓慢
Closed a year ago1
模型加载
Closed a year ago1
TypeError: 'NoneType' object is not subscriptable
Closed a year ago3
change dataset meet StopIteration
Closed a year ago
关于batchsize问题
Closed a year ago4
训练时，loss =nan
Closed a year ago5
loss很快降为0
Closed a year ago19
生成的模型，怎么转为huggleface格式呢
Closed a year ago2
请问这个错误，是transformer的版本问题吗？
Closed a year ago4