Giters

huangyangyi / TeCH

[3DV 2024] Official repo of "TeCH: Text-guided Reconstruction of Lifelike Clothed Humans"

Home Page:https://huangyangyi.github.io/TeCH/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

GPU memory limited

liustu opened this issue · comments

commented

Hi, would there another solution for "Step 3: Finetune Dreambooth model (minimal GPU memory requirement: 2x32G): with only a single 3090 GPU?

commented

We plan to test how TeCH works with more efficient DreamBooth finetune strategies, but you can try it out yourself first by following the examples in diffusers.

commented
commented

I was run step 3 in two RTX 3090 by using half precision and apex amp_backend
微信截图_20231110111818

commented

I was run step 3 in two RTX 3090 by using half precision and apex amp_backend 微信截图_20231110111818

Could you please show how to modify the code to implement the half precision training? I have tried to use apex amp in two GPU with 24GB each but it still goes out of memory.

commented

I was run step 3 in two RTX 3090 by using half precision and apex amp_backend 微信截图_20231110111818

Could you please show how to modify the code to implement the half precision training? I have tried to use apex amp in two GPU with 24GB each but it still goes out of memory.

add precision: 16 in v1-finetune_unfrozen.yaml then run step 3 you may get some error like RuntimeError: expected scalar type Float but found Half.
find error and fix it by add with torch.autocast("cuda"): in the top (note: the error always in forward or _forward)
I also change optimizer into 8 bit according to https://github.com/TimDettmers/bitsandbytes#requirements--installation
github