请问训练出现LastLoss:nan AvgLoss:nan,是怎么回事,要怎么处理,谢谢
zlszl opened this issue · comments
共122个图,配置如下:
Model:
CharSet: [' ', '8', '2', r, v, h, m, b, '5', w, '7', t, k, '6', y, p, '3', l,
q, x, a, e, f, n, s, '4']
ImageChannel: 1
ImageHeight: 64
ImageWidth: -1
Word: false
System:
Allow_Ext: [jpg, jpeg, png, bmp]
GPU: true
GPU_ID: 0
Path: E:\Val\images_login
Project: mlogin
Val: 0.03
Train:
BATCH_SIZE: 32
CNN: {NAME: ddddocr}
DROPOUT: 0.3
LR: 0.01
OPTIMIZER: SGD
SAVE_CHECKPOINTS_STEP: 2000
TARGET: {Accuracy: 0.97, Cost: 0.05, Epoch: 20}
TEST_BATCH_SIZE: 32
TEST_STEP: 1000
2022-07-12 00:38:15.250 | INFO | utils.train:start:108 - [2022-07-12-00_38_15] Epoch: 140500 Step: 421500 LastLoss: nan AvgLoss: nan Lr: 0.00015268545525806817