Hi, please help me

[ZzzTD]

Excuse me, I would like to create a binary classification and then create my own dataset in the ADE20K format, but after training, it will lose to nan. May I ask why?

iter: 100/160000, lr: 0.003454, eta: 8:47:42, time: 20.00, loss: nan, loss_pre: nan, loss_aux0: nan, loss_aux1: nan, loss_aux2: nan, loss_aux3: nan

[CoinCheung]

Hi,

Would you tell me the steps following which I can reproduce this phenomenon?

[ZzzTD]

Sorry, I have already solved this problem, but now the model will stop after 200 iterations, like this

iter: 100/150000, lr: 0.003454, eta: 1 day, 7:04:47, time: 75.39, loss: 39.9975, loss_pre: 16.9967, loss_aux0: 0.7748, loss_aux1: 1.0842, loss_aux2: 2.4893, loss_aux3: 18.6526
iter: 200/150000, lr: 0.004348, eta: 1 day, 6:27:07, time: 71.71, loss: 4.9907, loss_pre: 1.9791, loss_aux0: 0.5550, loss_aux1: 0.6058, loss_aux2: 0.7027, loss_aux3: 1.1480

save models to ./res\model_final.pth

evaluating the final model
compute single scale metrics
100%|██████████| 165/165 [00:45<00:00, 3.66it/s]
compute single scale crop metrics
100%|██████████| 165/165 [00:52<00:00, 3.12it/s]
compute multi scale flip metrics
100%|██████████| 165/165 [03:06<00:00, 1.13s/it]
compute multi scale flip crop metrics
100%|██████████| 165/165 [06:00<00:00, 2.19s/it]

eval results of f1 score metric:

| f1 score | ratio | ss | ssc | msf | msfc |
|------------+----------+----------+----------+----------+----------|
| cat 0 | 0.992426 | 0.99603 | 0.996075 | 0.996355 | 0.996389 |
| cat 1 | 0.007574 | 0.285909 | 0.27484 | 0.262318 | 0.252175 |
| macro_f1 | - | 0.64097 | 0.635457 | 0.629336 | 0.624282 |
| micro_f1 | - | 0.992105 | 0.992193 | 0.992746 | 0.992814 |

eval results of miou metric:

| iou | ratio | ss | ssc | msf | msfc |
|----------+----------+----------+----------+----------+----------|
| cat 0 | 0.992426 | 0.992092 | 0.992182 | 0.992737 | 0.992806 |
| cat 1 | 0.007574 | 0.1668 | 0.159313 | 0.150959 | 0.144279 |
| mious | - | 0.579446 | 0.575747 | 0.571848 | 0.568543 |
| fw_mious | - | 0.985842 | 0.985873 | 0.986361 | 0.986379 |

Process finished with exit code 0

and the congfig is bisenetv2_city.py:
cfg = dict(
model_type='bisenetv2',
n_cats=2,
num_aux_heads=4,
lr_start=5e-3,
weight_decay=5e-4,
warmup_iters=1000,
max_iter=150000,
dataset='CityScapes',
im_root=r'E:\Pythonfiles\BiSeNet-master\BiSeNet-master\datasets\cityscapes',
train_im_anns=r'E:\Pythonfiles\BiSeNet-master\BiSeNet-master\datasets\cityscapes\train.txt',
val_im_anns=r'E:\Pythonfiles\BiSeNet-master\BiSeNet-master\datasets\cityscapes\val.txt',
scales=[0.25, 2.],
cropsize=[512, 1024],
eval_crop=[1024, 1024],
eval_scales=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
ims_per_gpu=8,
eval_ims_per_gpu=2,
use_fp16=True,
use_sync_bn=False,
respth='./res',
)

Could you please help me see why this is happening