Giters

andyzeng / visual-pushing-grasping

Train robotic agents to learn to plan pushing and grasping actions for manipulation with deep reinforcement learning.

Home Page:http://vpg.cs.princeton.edu/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

cuda out of memory

luliangliangbeifeng opened this issue · comments

commented

I have the problem that the terminal show that CUDA out of menmory when I run "python main.py --is_sim --method 'reactive' --experience_replay --save_visualizations". However everything is ok when I run "python main.py --is_sim --push_rewards --experience_replay --explore_rate_decay --save_visualizations". Do you know this is why?
Thank you!
image

commented

You're having the same issue as here. I am not sure what gpu specifications you are running it on, but theres a chance yours cannot handle the backpropagation. The only way I found to run the reactive net was to comment/edit the lines they described in this link.
#70

commented

I have an interesting observation. You only encounter this problem when using the argument "--method 'reactive'", which only changes the network model. So I think the root of this problem is not lack of GPU memory (I use 3080 with 12GB memory, equal to that of Titan X). There might be something wrong with the reactive net model.

commented

It's a memory leak problem. The GPU memory usage increases per iteration. So no better how large memory your device has, the program will Eat It Up. I think it has something to do with the computation graph of the reactive_net, which caused the memory leak issue.