OOM appears when training the model

Question

OOM appears when training the model

jacypne opened this issue 5 years ago · comments

Hi! I have a problom like the issue #18 , and I tried with reduced batch size and dimensions as you answered. I set the batch equal one but it also had same problom, OOM. My graphics card is Nvidia GeForce RTX 2080ti (11G), why is the memory superimposed when the program is running? Moreover, when I set the lstm_dim equal 64 and the batch equal 32, I got result auc: 0.397, it didn't match your paper auc: 0.416 on Riedel. Could you please give any help? The error is shown as follow:

Shikhar Vashishth · Answer 1 · Tue May 14 2019 00:12:06 GMT+0800 (China Standard Time)

Hi, can you try running it on CPU to check whether it is working or not?
Use larger LSTM dimension like 128 or 192 for matching the reported performance.

ejld · Answer 2 · Tue May 14 2019 12:30:10 GMT+0800 (China Standard Time)

@svjan5 Hi, I am having a same error on Nvidia 1080Ti, it does work on CPU though very slow. Whatever batchsize I try it always runs out of memory. I wonder how we can reproduce the results under these hardware settings?

Shikhar Vashishth · Answer 3 · Tue May 14 2019 12:32:55 GMT+0800 (China Standard Time)

I have trained all the models on 1080Ti only. It used to take around 8gb of memory. I think the new version of Tensorflow might be demanding some more space. Can you try experimenting with 1.8 version?

ejld · Answer 4 · Tue May 14 2019 12:42:40 GMT+0800 (China Standard Time)

I have trained all the models on 1080Ti only. It used to take around 8gb of memory. I think the new version of Tensorflow might be demanding some more space. Can you try experimenting with 1.8 version?

Thanks for the hint, I will try and let you know how it goes. 8gb should be alright, we have the same card with 11gb I believe. The strange thing is that it can always start training normally, with reasonable memory usage, but after some random steps the oom error appears. It seems the cuda memory is stacking up during the training process.

ejld · Answer 5 · Tue May 14 2019 14:14:15 GMT+0800 (China Standard Time)

I have trained all the models on 1080Ti only. It used to take around 8gb of memory. I think the new version of Tensorflow might be demanding some more space. Can you try experimenting with 1.8 version?

Hi, we have tried using 1.8, still have the same problem, do you have any other suggestions? could you post a list of the packages you use so we can check?

Shikhar Vashishth · Answer 6 · Sun Jun 23 2019 22:09:19 GMT+0800 (China Standard Time)

I am sorry for the late reply. I am able to still run my code without any OOM error. You can look at the output of pip freeze here: https://codebunk.com/b/694343322/