Training GPU requirement

Question

Training GPU requirement

Sm0kyWu opened this issue 2 months ago · comments

Hi! Thanks for the amazing code!

I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.

Could you please kindly share some information about training?

Thanks!

Zhengyan Tong · Answer 1 · Wed Jun 12 2024 10:39:34 GMT+0800 (China Standard Time)

Hi! Thanks for the amazing code!

I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.

Could you please kindly share some information about training?

Thanks!

8x80GB deepspeed zero2

房森 · Answer 2 · Sun Jun 23 2024 16:52:32 GMT+0800 (China Standard Time)

@Sm0kyWu If you use the training code for the mole thread, does it need some modification and approximately how much extra time will it take?

Sm0kyWu · Answer 3 · Sun Jun 23 2024 17:21:15 GMT+0800 (China Standard Time)

@FangSen9000 I haven't tried to train the Moore code. For Musepose you can directly use the training script from Moore. Single 40G A100 takes around 40 hours for 10000 steps of stage 1 (bs 6, 768x768, deepspeed zero2).