OOM in pretraining
hgzjy25 opened this issue · comments
I tried to pretrain HERO mode from scratch in HowTo100M and TV datasets, and the code worked well at the begining, but crashed after thousands of iterations. I found that the memory usage was growing in training and finally out of memory. Have you met this problem?
I also encounter the same problem. @linjieli222
I have received similar reports about this issue. However, we did not met the same issue during our experiments. You may need to search online for potential solutions, sorry for any inconvenience. If you do find a solution, please also come back and post it here helping other people in need.
One potential direction, check if the memory increasing is due to caching. If so, you can force to clean the cache periodically.