Diffusion loss not decreasing
aniketp02 opened this issue · comments
Hi,
I have trained the GradTTS model on the Indian accent English dataset, and the results are pretty awesome.
Looking at the logs, I was startled to see that the diffusion loss throughout the training was not decreasing unlike other losses, and was also fluctuating a lot. Can anyone explain to me why this is the case and if the diffusion loss fluctuates so much why is it used in the total loss calculation?
I have attached my tensorboard outputs.
@aniketp02 Hi! All 3 losses are must-have to train the model properly. What you have observed about diffusion loss is a normal behaviour, which we discussed in Section 4 of our Grad-TTS paper.
The denoising score matching objective we want to minimize to train a diffusion model is the integral
Combining these facts we get such diffusion loss behavior. Nonetheless, that doesn't mean it is unnecessary to optimize, moreover it is crucial. Otherwise, Grad-TTS diffusion decoder would produce just noise. Finally, diffusion loss does the job well if we check the energy function it corresponds to. Look at this issue: #9.