MediaBrain-SJTU / LED

[CVPR2023] Leapfrog Diffusion Model for Stochastic Trajectory Prediction

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Out of memory issue while running ''def _test_single_epoch(self):'' function

daviddaiiiii opened this issue · comments

Hi there, I'm currently trying to reproduce the training and evaluation process but have run into an issue with the "def _test_single_epoch(self):" function causing an OOM error.

I have this issue running on a 24G RTX4090 GPU, it seems that this is not due to hardware limitations.

Do you have any suggestions on how to modify the function and release some memory during the evaluation?

########################################################
Trying to train from scratch
image

########################################################
modify
image
get
image
confirm the issue is due to ''def _test_single_epoch(self):'' function
########################################################
solely evaluating with pre-trained weight
image