Grad scale is too small

[hoangtm-aimesoft]

Hi all. I'm trying to train streaming zipformer on reazonspeech dataset (100h) with reazonspeech recipe and I got this error:

Here is my training args

Can someone help me ? Thank you very much !

[Fangjun Kuang]

@JinZr Could you have a look?

[zr_jin]

Hi, please update your local cloned repo to the latest master branch and try again.

Thanks!

[Trần Minh Hoàng (Artorias)]

@JinZr Thank you for your reply. I have pulled the latest master branch but I cannot see any update on the reazonspeech recipe. Can you explain it ?

[zr_jin]

hi, if so, please try reducing base_lr by 0.005 (i.e. from 0.045 to 0.04) and see if it works. could you also provide us with the information of your computational resources you’ve been working with? Best Regards Jin

…

On Sat, 22 Jun 2024 at 03:21 Trần Minh Hoàng (Artorias) < ***@***.***> wrote: @JinZr <https://github.com/JinZr> Thank you for your reply. I have pulled the latest master branch but I cannot see any update on the reazonspeech recipe. Can you explain it ? — Reply to this email directly, view it on GitHub <#1661 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOON42BG2CYXOZFPQOUFKP3ZIR4LBAVCNFSM6AAAAABJVSVXQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTGMZDKNJQGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Daniel Povey]

Next time try pasting more of the error. That error appears if the model diverges but without seeing more output it can be hard to give more precise help other than "reduce the LR".

[Trần Minh Hoàng (Artorias)]

@JinZr I'm working with NVIDIA GeForce RTX 3090 with 24GB VRAM

[Xuanwu Yun]

Yes, I am also training a streaming model on ReazonSpeech. I trained with 3k hours of data using a single Nvidia 3090 GPU, with a learning rate of 0.025. The training proceeds normally and the convergence is quite good, with each epoch taking approximately 4 hours.