Out of memory with RTX3090

Question

Out of memory with RTX3090

PyxAI opened this issue 2 years ago · comments

Hi,
I'm trying to train gpt2xl, but keep getting OOM, even when I set batch size to 1 and gradient_accumulation to 8\16\512, contigous_gradients false and allgather_bucket_size \ reduce_bucket_size 2e2.
I can see in nvidia-smi that I'm only reaching half the memory capacity - around 12GB
My system is as stated - 3090 with 24GB memory
80 GB RAM
5600x cpu if that matters
running WSL2 on windows 10
Thanks.

Omri Algazi · Answer 1 · Sat Jan 15 2022 01:40:20 GMT+0800 (China Standard Time)

So working with WSL is just a no-go
I installed dual boot ubuntu and now the problem disappeared

Brandon Koerner · Answer 2 · Tue May 10 2022 08:58:40 GMT+0800 (China Standard Time)

dual boot only, huh... that sucks. I was really hoping I could use this on win10 or wsl(2)

Omri Algazi · Answer 3 · Wed May 18 2022 01:38:01 GMT+0800 (China Standard Time)

I was however, able to run the model under WSL2 windows 11
Didn't check training, it's worth a shot @ReewassSquared

Usman Ahmad · Answer 4 · Thu Sep 08 2022 17:50:23 GMT+0800 (China Standard Time)

Hi @PyxAI . Which ubuntu version did you run this code on?