Giters

nasa / pretrained-microscopy-models

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Training not converge on EBC subsets

linjiangya opened this issue · comments

commented

Thank you so much for your kind reply before. I have successfully reproduced results on superalloyed subsets (Super1-4). However, when it comes to EBC subsets. There are still some problems:

  1. The format of annotation files in EBC subsets seem to be different from those in Super1-4. EBC's annotations are already within the range [0-1] so the logic in io.py might need to be changed. (here, and here). Otherwise the visualization of GT will be a completely purple image like this:
    image

  2. I simply followed the example, and comment the lines and changed the path from Super1 to EBC1 to let it work for EBC1. The GT visualization seems to be normal now, however, the loss value didn't converge.

image

train:   0%|          | 0/3 [00:02<?, ?it/s, DiceBCELoss - 0.7874, iou_score - 0.08617]
train:  33%|███▎      | 1/3 [00:02<00:05,  3.00s/it, DiceBCELoss - 0.7874, iou_score - 0.08617]
train:  33%|███▎      | 1/3 [00:03<00:05,  3.00s/it, DiceBCELoss - 0.7894, iou_score - 0.06694]
train:  67%|██████▋   | 2/3 [00:03<00:01,  1.56s/it, DiceBCELoss - 0.7894, iou_score - 0.06694]
train:  67%|██████▋   | 2/3 [00:04<00:01,  1.56s/it, DiceBCELoss - 0.7873, iou_score - 0.04933]
train: 100%|██████████| 3/3 [00:04<00:00,  1.08s/it, DiceBCELoss - 0.7873, iou_score - 0.04933]
train: 100%|██████████| 3/3 [00:04<00:00,  1.36s/it, DiceBCELoss - 0.7873, iou_score - 0.04933]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7657, iou_score - 0.1046]
valid: 100%|██████████| 1/1 [00:00<00:00,  7.96it/s, DiceBCELoss - 0.7657, iou_score - 0.1046]
valid: 100%|██████████| 1/1 [00:00<00:00,  7.95it/s, DiceBCELoss - 0.7657, iou_score - 0.1046]
Best model saved!

Epoch: 1, lr: 0.00020000, time: 16.11 seconds, patience step: 0, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.774, iou_score - 0.02094]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.87it/s, DiceBCELoss - 0.774, iou_score - 0.02094]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.87it/s, DiceBCELoss - 0.7784, iou_score - 0.01238]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.07it/s, DiceBCELoss - 0.7784, iou_score - 0.01238]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.07it/s, DiceBCELoss - 0.7864, iou_score - 0.008447]
train: 100%|██████████| 3/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7864, iou_score - 0.008447]
train: 100%|██████████| 3/3 [00:01<00:00,  2.02it/s, DiceBCELoss - 0.7864, iou_score - 0.008447]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7639, iou_score - 0.06062]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.01it/s, DiceBCELoss - 0.7639, iou_score - 0.06062]

Epoch: 2, lr: 0.00020000, time: 7.93 seconds, patience step: 1, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.8061, iou_score - 0.002352]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.70it/s, DiceBCELoss - 0.8061, iou_score - 0.002352]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.70it/s, DiceBCELoss - 0.7943, iou_score - 0.001258]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.00it/s, DiceBCELoss - 0.7943, iou_score - 0.001258]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.00it/s, DiceBCELoss - 0.7862, iou_score - 0.002372]
train: 100%|██████████| 3/3 [00:01<00:00,  2.06it/s, DiceBCELoss - 0.7862, iou_score - 0.002372]
train: 100%|██████████| 3/3 [00:01<00:00,  2.01it/s, DiceBCELoss - 0.7862, iou_score - 0.002372]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7633, iou_score - 0.009991]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.80it/s, DiceBCELoss - 0.7633, iou_score - 0.009991]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.78it/s, DiceBCELoss - 0.7633, iou_score - 0.009991]

Epoch: 3, lr: 0.00020000, time: 7.82 seconds, patience step: 2, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7677, iou_score - 0.0001071]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.63it/s, DiceBCELoss - 0.7677, iou_score - 0.0001071]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.63it/s, DiceBCELoss - 0.7757, iou_score - 7.909e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.86it/s, DiceBCELoss - 0.7757, iou_score - 7.909e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.86it/s, DiceBCELoss - 0.7862, iou_score - 0.0001844]
train: 100%|██████████| 3/3 [00:01<00:00,  1.88it/s, DiceBCELoss - 0.7862, iou_score - 0.0001844]
train: 100%|██████████| 3/3 [00:01<00:00,  1.85it/s, DiceBCELoss - 0.7862, iou_score - 0.0001844]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7632, iou_score - 0.0001064]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.05it/s, DiceBCELoss - 0.7632, iou_score - 0.0001064]

Epoch: 4, lr: 0.00020000, time: 7.93 seconds, patience step: 3, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.8111, iou_score - 2.473e-05]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.72it/s, DiceBCELoss - 0.8111, iou_score - 2.473e-05]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.72it/s, DiceBCELoss - 0.7961, iou_score - 6.18e-05] 
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.88it/s, DiceBCELoss - 0.7961, iou_score - 6.18e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.88it/s, DiceBCELoss - 0.7862, iou_score - 6.292e-05]
train: 100%|██████████| 3/3 [00:01<00:00,  1.92it/s, DiceBCELoss - 0.7862, iou_score - 6.292e-05]
train: 100%|██████████| 3/3 [00:01<00:00,  1.89it/s, DiceBCELoss - 0.7862, iou_score - 6.292e-05]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.524e-05]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.00it/s, DiceBCELoss - 0.7631, iou_score - 1.524e-05]

Epoch: 5, lr: 0.00020000, time: 7.93 seconds, patience step: 4, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7616, iou_score - 3.223e-05]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.80it/s, DiceBCELoss - 0.7616, iou_score - 3.223e-05]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.80it/s, DiceBCELoss - 0.7715, iou_score - 0.001048] 
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.05it/s, DiceBCELoss - 0.7715, iou_score - 0.001048]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.05it/s, DiceBCELoss - 0.7864, iou_score - 0.0006984]
train: 100%|██████████| 3/3 [00:01<00:00,  2.14it/s, DiceBCELoss - 0.7864, iou_score - 0.0006984]
train: 100%|██████████| 3/3 [00:01<00:00,  2.08it/s, DiceBCELoss - 0.7864, iou_score - 0.0006984]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.715e-05]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.72it/s, DiceBCELoss - 0.7631, iou_score - 1.715e-05]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.70it/s, DiceBCELoss - 0.7631, iou_score - 1.715e-05]

Epoch: 6, lr: 0.00020000, time: 7.74 seconds, patience step: 5, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7808, iou_score - 1.088e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.81it/s, DiceBCELoss - 0.7808, iou_score - 1.088e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.81it/s, DiceBCELoss - 0.7841, iou_score - 1.121e-13]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.08it/s, DiceBCELoss - 0.7841, iou_score - 1.121e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.08it/s, DiceBCELoss - 0.7857, iou_score - 1.143e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.18it/s, DiceBCELoss - 0.7857, iou_score - 1.143e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.12it/s, DiceBCELoss - 0.7857, iou_score - 1.143e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.22it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 7, lr: 0.00020000, time: 7.84 seconds, patience step: 6, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7807, iou_score - 1.096e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.76it/s, DiceBCELoss - 0.7807, iou_score - 1.096e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.76it/s, DiceBCELoss - 0.7749, iou_score - 1.188e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.96it/s, DiceBCELoss - 0.7749, iou_score - 1.188e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.96it/s, DiceBCELoss - 0.7861, iou_score - 0.0005052]
train: 100%|██████████| 3/3 [00:01<00:00,  2.08it/s, DiceBCELoss - 0.7861, iou_score - 0.0005052]
train: 100%|██████████| 3/3 [00:01<00:00,  2.02it/s, DiceBCELoss - 0.7861, iou_score - 0.0005052]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.85it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.83it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 8, lr: 0.00020000, time: 7.42 seconds, patience step: 7, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7876, iou_score - 1.165e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.78it/s, DiceBCELoss - 0.7876, iou_score - 1.165e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.78it/s, DiceBCELoss - 0.7843, iou_score - 1.13e-13] 
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7843, iou_score - 1.13e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7857, iou_score - 2.365e-06]
train: 100%|██████████| 3/3 [00:01<00:00,  2.07it/s, DiceBCELoss - 0.7857, iou_score - 2.365e-06]
train: 100%|██████████| 3/3 [00:01<00:00,  2.02it/s, DiceBCELoss - 0.7857, iou_score - 2.365e-06]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.84it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.82it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 9, lr: 0.00020000, time: 7.84 seconds, patience step: 8, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7819, iou_score - 1.107e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.83it/s, DiceBCELoss - 0.7819, iou_score - 1.107e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.83it/s, DiceBCELoss - 0.7904, iou_score - 1.209e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7904, iou_score - 1.209e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7858, iou_score - 1.16e-13] 
train: 100%|██████████| 3/3 [00:01<00:00,  2.15it/s, DiceBCELoss - 0.7858, iou_score - 1.16e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.09it/s, DiceBCELoss - 0.7858, iou_score - 1.16e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.13it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 10, lr: 0.00020000, time: 8.09 seconds, patience step: 9, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7929, iou_score - 1.237e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.89it/s, DiceBCELoss - 0.7929, iou_score - 1.237e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.89it/s, DiceBCELoss - 0.7824, iou_score - 1.125e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7824, iou_score - 1.125e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7858, iou_score - 1.159e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  1.97it/s, DiceBCELoss - 0.7858, iou_score - 1.159e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  1.97it/s, DiceBCELoss - 0.7858, iou_score - 1.159e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.92it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.90it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 11, lr: 0.00020000, time: 7.98 seconds, patience step: 10, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.8117, iou_score - 1.509e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.78it/s, DiceBCELoss - 0.8117, iou_score - 1.509e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.78it/s, DiceBCELoss - 0.7799, iou_score - 0.0002374]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.04it/s, DiceBCELoss - 0.7799, iou_score - 0.0002374]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.04it/s, DiceBCELoss - 0.7867, iou_score - 0.0001583]
train: 100%|██████████| 3/3 [00:01<00:00,  2.26it/s, DiceBCELoss - 0.7867, iou_score - 0.0001583]
train: 100%|██████████| 3/3 [00:01<00:00,  2.16it/s, DiceBCELoss - 0.7867, iou_score - 0.0001583]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.24it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.22it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 12, lr: 0.00020000, time: 6.91 seconds, patience step: 11, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.794, iou_score - 1.252e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.99it/s, DiceBCELoss - 0.794, iou_score - 1.252e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.99it/s, DiceBCELoss - 0.7798, iou_score - 6.275e-06]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.03it/s, DiceBCELoss - 0.7798, iou_score - 6.275e-06]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7859, iou_score - 4.183e-06]
train: 100%|██████████| 3/3 [00:01<00:00,  2.15it/s, DiceBCELoss - 0.7859, iou_score - 4.183e-06]
train: 100%|██████████| 3/3 [00:01<00:00,  2.11it/s, DiceBCELoss - 0.7859, iou_score - 4.183e-06]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.96it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.93it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 13, lr: 0.00020000, time: 7.74 seconds, patience step: 12, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7984, iou_score - 1.309e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.93it/s, DiceBCELoss - 0.7984, iou_score - 1.309e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.93it/s, DiceBCELoss - 0.7794, iou_score - 1.119e-13]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.04it/s, DiceBCELoss - 0.7794, iou_score - 1.119e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.04it/s, DiceBCELoss - 0.7861, iou_score - 1.186e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7861, iou_score - 1.186e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.02it/s, DiceBCELoss - 0.7861, iou_score - 1.186e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.29it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 14, lr: 0.00020000, time: 7.82 seconds, patience step: 13, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7919, iou_score - 1.224e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.93it/s, DiceBCELoss - 0.7919, iou_score - 1.224e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.93it/s, DiceBCELoss - 0.7904, iou_score - 1.208e-13]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.14it/s, DiceBCELoss - 0.7904, iou_score - 1.208e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.14it/s, DiceBCELoss - 0.7857, iou_score - 1.158e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.11it/s, DiceBCELoss - 0.7857, iou_score - 1.158e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.10it/s, DiceBCELoss - 0.7857, iou_score - 1.158e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  8.45it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  8.44it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 15, lr: 0.00020000, time: 7.83 seconds, patience step: 14, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7767, iou_score - 1.063e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.97it/s, DiceBCELoss - 0.7767, iou_score - 1.063e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.97it/s, DiceBCELoss - 0.791, iou_score - 1.233e-13] 
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.10it/s, DiceBCELoss - 0.791, iou_score - 1.233e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.10it/s, DiceBCELoss - 0.7859, iou_score - 1.175e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.15it/s, DiceBCELoss - 0.7859, iou_score - 1.175e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.12it/s, DiceBCELoss - 0.7859, iou_score - 1.175e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.73it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.71it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 16, lr: 0.00020000, time: 7.21 seconds, patience step: 15, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7983, iou_score - 0.001186]
train:  33%|███▎      | 1/3 [00:00<00:00,  2.06it/s, DiceBCELoss - 0.7983, iou_score - 0.001186]
train:  33%|███▎      | 1/3 [00:00<00:00,  2.06it/s, DiceBCELoss - 0.7828, iou_score - 0.0005932]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.03it/s, DiceBCELoss - 0.7828, iou_score - 0.0005932]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7859, iou_score - 0.0003955]
train: 100%|██████████| 3/3 [00:01<00:00,  1.95it/s, DiceBCELoss - 0.7859, iou_score - 0.0003955]
train: 100%|██████████| 3/3 [00:01<00:00,  1.97it/s, DiceBCELoss - 0.7859, iou_score - 0.0003955]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.65it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.63it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 17, lr: 0.00020000, time: 7.76 seconds, patience step: 16, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7952, iou_score - 1.269e-13]
train:  33%|███▎      | 1/3 [00:00<00:00,  2.00it/s, DiceBCELoss - 0.7952, iou_score - 1.269e-13]
train:  33%|███▎      | 1/3 [00:00<00:00,  2.00it/s, DiceBCELoss - 0.7895, iou_score - 1.204e-13]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.03it/s, DiceBCELoss - 0.7895, iou_score - 1.204e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.03it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.13it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.10it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.05it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 18, lr: 0.00020000, time: 7.74 seconds, patience step: 17, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7719, iou_score - 1.021e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.82it/s, DiceBCELoss - 0.7719, iou_score - 1.021e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.82it/s, DiceBCELoss - 0.7795, iou_score - 5.099e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7795, iou_score - 5.099e-05]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7858, iou_score - 3.399e-05]
train: 100%|██████████| 3/3 [00:01<00:00,  2.00it/s, DiceBCELoss - 0.7858, iou_score - 3.399e-05]
train: 100%|██████████| 3/3 [00:01<00:00,  1.98it/s, DiceBCELoss - 0.7858, iou_score - 3.399e-05]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.97it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.95it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 19, lr: 0.00020000, time: 7.98 seconds, patience step: 18, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.808, iou_score - 1.454e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.79it/s, DiceBCELoss - 0.808, iou_score - 1.454e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.79it/s, DiceBCELoss - 0.7976, iou_score - 1.314e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.92it/s, DiceBCELoss - 0.7976, iou_score - 1.314e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  1.92it/s, DiceBCELoss - 0.7861, iou_score - 1.192e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  1.94it/s, DiceBCELoss - 0.7861, iou_score - 1.192e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  1.92it/s, DiceBCELoss - 0.7861, iou_score - 1.192e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00, 10.08it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 20, lr: 0.00020000, time: 7.22 seconds, patience step: 19, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7733, iou_score - 1.036e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.88it/s, DiceBCELoss - 0.7733, iou_score - 1.036e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.88it/s, DiceBCELoss - 0.7835, iou_score - 1.144e-13]
train:  67%|██████▋   | 2/3 [00:00<00:00,  2.06it/s, DiceBCELoss - 0.7835, iou_score - 1.144e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.06it/s, DiceBCELoss - 0.7857, iou_score - 1.165e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.06it/s, DiceBCELoss - 0.7857, iou_score - 1.165e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.04it/s, DiceBCELoss - 0.7857, iou_score - 1.165e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.76it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.71it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 21, lr: 0.00020000, time: 7.81 seconds, patience step: 20, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7758, iou_score - 1.057e-13]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.74it/s, DiceBCELoss - 0.7758, iou_score - 1.057e-13]
train:  33%|███▎      | 1/3 [00:01<00:01,  1.74it/s, DiceBCELoss - 0.7837, iou_score - 1.143e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.01it/s, DiceBCELoss - 0.7837, iou_score - 1.143e-13]
train:  67%|██████▋   | 2/3 [00:01<00:00,  2.01it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  2.00it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]
train: 100%|██████████| 3/3 [00:01<00:00,  1.97it/s, DiceBCELoss - 0.7857, iou_score - 1.162e-13]

valid:   0%|          | 0/1 [00:00<?, ?it/s]
valid:   0%|          | 0/1 [00:00<?, ?it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.85it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]
valid: 100%|██████████| 1/1 [00:00<00:00,  9.83it/s, DiceBCELoss - 0.7631, iou_score - 1.905e-13]

Epoch: 22, lr: 0.00020000, time: 7.90 seconds, patience step: 21, best iou: 0.1046

train:   0%|          | 0/3 [00:00<?, ?it/s]
train:   0%|          | 0/3 [00:00<?, ?it/s, DiceBCELoss - 0.7687, iou_score - 9.955e-14]
train:  33%|███▎      | 1/3 [00:00<00:01,  1.90it/s, DiceBCELoss - 0.7687, iou_score - 9.955e-14]
commented

Hi,

The only logic that should need to be changed is the class_values and the file paths. The second logic you commented out is for the different mask channels, not the different annotations. (the lines)

Try changing the class values to something like:
class_values = {'background': [0],
'oxide': [1],
'cracks' : [2]}

Cell 7 "Visualize Datasets" should show the annotated masks.

You can also check out the binary_segmentation example for the EBC dataset.

Josh

commented

class_values = {'background': [0],
'oxide': [1],
'cracks' : [2]}

Thank you for your prompt reply. I have been following your instruction and only change the class values (the only other change I've made is to resize the EBC1 images & masks to [512,512] resolution because of GPU OOM issue on 40GB A100). This the new results on the EBC1 test set (040617#2_S127(2)0015.tif): train.log

image
Now it generally works! :) But it's just that the 'cracks' are not detected. Thank you again for your contribution!

commented

I'm glad its generally working! Yeah the cracks are hard and to get it to work on my end I needed more training data than I provided in this repository. I decided to keep the labels in the benchmark data to provide a way to test future improved methods. I didn't really use the cracks in the MicroNet paper associated with this repo. (In the future I'll be publishing a paper based on another project that did much more detailed analysis of EBC data.)