how to train for 30b or 65b on multiple GPU（4x80G）

Question

lishangjin opened this issue a year ago · comments

lishangjin · Answer 1 · Mon Mar 27 2023 13:47:17 GMT+0800 (China Standard Time)

https://zhuanlan.zhihu.com/p/616853024 提及：依赖的lora中引入的transformers支持多GPU存在问题。可以使用https://github.com/kooshi/alpaca-lora/tree/llama-parallelism llama-parallelism分支替换