Training GPU requirement
Sm0kyWu opened this issue · comments
Hi! Thanks for the amazing code!
I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.
Could you please kindly share some information about training?
Thanks!
Hi! Thanks for the amazing code!
I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.
Could you please kindly share some information about training?
Thanks!
8x80GB deepspeed zero2
@Sm0kyWu If you use the training code for the mole thread, does it need some modification and approximately how much extra time will it take?
@FangSen9000 I haven't tried to train the Moore code. For Musepose you can directly use the training script from Moore. Single 40G A100 takes around 40 hours for 10000 steps of stage 1 (bs 6, 768x768, deepspeed zero2).