Slow Training Speed on LRS2 Dataset with 4x RTX 4090 GPUs,(train_video_renderer.py)
Kiri0824 opened this issue · comments
I attempted to run train_video_renderer.py on the LRS2 dataset using four RTX 4090 GPUs, but the training speed is exceptionally slow. In a previous issue, I noticed that the author suggested running approximately 300 epochs for optimal results. However, the speed I'm experiencing is much lower than expected. Does anyone has same issue?
BTW, #12 , i see the author said: We stop training near 300 epochs, where FID is around 19, eval_gen_loss is around 7, and eval_warp_loss is around 11. But this training uses ref_N=3.
Here's my loss:
I didn't train 25 epoch, so others loss didn't decrease. But the eval_warp_loss is a little low, is that normal😢
it's normal. i fix it😊
Can you share how you fixed it?
Can you share how you fixed it?
sorry for a long time to reply, its just normal, just wait for a few days :)