Bohao-Lee / CME

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

occur the NAN when training the Net using the pascal voc

SamXiaosheng opened this issue · comments

commented

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000
coord_mask: tensor(27.0292, device='cuda:0')
80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan
coord_mask: tensor(23.9451, device='cuda:0')
96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan
coord_mask: tensor(nan, device='cuda:0')
112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan
coord_mask: tensor(nan, device='cuda:0')
128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan
coord_mask: tensor(nan, device='cuda:0')
144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000 coord_mask: tensor(27.0292, device='cuda:0') 80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan coord_mask: tensor(23.9451, device='cuda:0') 96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

我也遇到了同样的问题,我想问一下这个问题你解决了吗?

64: nGT 42, recall 1, proposals 35613, loss: x 27.029242, y 28.630236, w 177730.062500, h 1531226.000000, conf 2786.200684, cls 108.162247, class_contrast 0.000000, total 1711906.125000 coord_mask: tensor(27.0292, device='cuda:0') 80: nGT 40, recall 13, proposals 39993, loss: x 23.945053, y 22.210649, w 2029862.625000, h 16542059.000000, conf nan, cls 111.281578, class_contrast 0.000000, total nan coord_mask: tensor(23.9451, device='cuda:0') 96: nGT 30, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 112: nGT 36, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 128: nGT 39, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan coord_mask: tensor(nan, device='cuda:0') 144: nGT 54, recall 0, proposals 0, loss: x nan, y nan, w nan, h nan, conf nan, cls nan, class_contrast nan, total nan

我也遇到了同样的问题,我想问一下这个问题你解决了吗?

[降低学习率试试,