About models training

[Teli Ma]

Thank you for your great work!

When I trained the model from scratch, I encountered that the training stucks like this:
.

The GPU utilization is MAX, but the GPU memory is very low.

So is it normal, or have you encountered the same situation?

Thanks.

[Ankit Goyal]

Hi,

Thanks for your interest in our work. I haven't seen this issue at my end so not sure what is going on. I would suggest putting some print statements at various locations in the train.py script to debug the issue.

Best,
Ankit

[Teli Ma]

Hi,

Thank you for your reply. I have solved the problem after changing the docker environment. I will close the issue.

Best,
Teli