Loss: Nan

Question

Loss: Nan

MiTeng0215 opened this issue 5 years ago · comments

用的是SynthText数据集，训练时出现：

RankWarning: Polyfit may be poorly conditioned 
edge = fit_line([p0[0], p1[0]], [p0[1], p1[1]])

上面的警告不影响训练，训练一段时间后loss出现Nan值。我并没有执行build.sh这个程序，因为学校集群环境中没有NVCC。看其他问题中提到好像也不需要编译它。
希望问题能得到解决。

MiTeng0215 · Answer 1 · Thu Jan 16 2020 21:41:35 GMT+0800 (China Standard Time)

Loss：Nan

niloofarAzari · Answer 2 · Mon Jan 20 2020 23:08:21 GMT+0800 (China Standard Time)

@MiTeng0215 i have the same problem!
Could you solve it? and how if yes?

MiTeng0215 · Answer 3 · Mon Jan 20 2020 23:16:02 GMT+0800 (China Standard Time)

@MiTeng0215 i have the same problem!
Could you solve it? and how if yes?

No, I'm looking at the code details of this project. You can add my WeChat and let's discuss it together.

MiTeng0215 · Answer 4 · Mon Jan 20 2020 23:17:15 GMT+0800 (China Standard Time)

@MiTeng0215 i have the same problem!
Could you solve it? and how if yes?

No, I'm looking at the code details of this project. You can add my WeChat and let's discuss it together.

My WeChat is Mteng215.

京局京段蓝白猪 · Answer 5 · Fri Jul 03 2020 15:29:26 GMT+0800 (China Standard Time)

Maybe CRNN's ctcloss function problem.
You can use pytorch==1.1.0 and warp-ctc or refer to this link.

mxin262 · Answer 6 · Fri Nov 20 2020 11:52:54 GMT+0800 (China Standard Time)

@MiTeng0215 你好，请问你解决这个问题了吗

MiTeng0215 · Answer 7 · Mon Apr 19 2021 16:29:38 GMT+0800 (China Standard Time)

@MiTeng0215 i have the same problem!
Could you solve it? and how if yes?

No, I'm looking at the code details of this project. You can add my WeChat and let's discuss it together.

My WeChat is Mteng215.

MiTeng0215 · Answer 8 · Mon Apr 19 2021 16:29:50 GMT+0800 (China Standard Time)

问题太久了,大概是因为ctc引起的问题,google一下ctc引起loss nan的问题,应该很好找到答案.这个项目我没有继续做下去,最终效果我也没有呈现.所以,我准备关闭问题了,也不要加我微信了,谢谢.