WangRongsheng / Aurora

🐳 Aurora is a [Chinese Version] MoE model. Aurora is a further work based on Mixtral-8x7B, which activates the chat capability of the model's Chinese open domain.

https://arxiv.org/abs/2312.14557

有没有多gpu推理的脚本?一张3090刚好放不下

wonder-hy opened this issue 4 months ago · comments

fredo commented 4 months ago

试了下用vllm多卡推理,没有成功...
在这试着问下
谢谢

WangRongsheng commented 4 months ago

嗨,

推荐您使用天宫云镜像一键部署使用：

镜像：https://www.xiangongyun.com/image/detail/bbcdc87e-20de-561c-a298-4323cf2f6fb2
视频教程：https://www.bilibili.com/video/BV1xw411E7To/

如果您需要在单机多卡上运行该模型，请下载app.py，并以：

CUDA_VISIBLE_DEVICES=0,1,.. python app.py