Loss not decreasing when training outside of script
plehman2000 opened this issue · comments
After running
python -m segm.scripts.prepare_ade20k $DATASET
python -m segm.train --log-dir seg_tiny_mask --dataset ade20k \ --backbone vit_tiny_patch16_384 --decoder mask_transformer
The training script runs without error, but the loss is not decreasing:
Upon further examination, when I isolated your segmenter class, I found that the parameters after the decoders transformer step do not seem to be updating.
See my note in the comment below for clarification. Any help is appreciated! I've been stuck for quite a while on this
(from decoder.py, MaskTransformer())
For clarification, I see that the average loss is decreasing, but is there any reason that making a minimum working example like this would not yield a changing loss?
MWE
Results
Unedited model
Returning tensor immediately after self.decoder_norm() step as the model output yields an updating loss
Are there any special considerations to be made when training this model outside of the training script?
How did you fix the code ?
I put my own data set into the ade20k folder and used "python -m segm.train --log-dir seg_tiny_mask --dataset ade20k
--backbone vit_tiny_patch16_384 --decoder mask_transformer" for training. I set data.ade20k.py as follow , to detection oil.
class ADE20KSegmentation(BaseMMSeg):
def __init__(self, image_size, crop_size, split, **kwargs):
super().__init__(
image_size,
crop_size,
split,
ADE20K_CONFIG_PATH,
**kwargs,
)
self.names, self.colors = utils.dataset_cat_description(ADE20K_CATS_PATH)
self.n_cls = 2
self.ignore_label = 255
self.reduce_zero_label =False
How is the.yml file set up in data?My data set has only one class of objects and backgrounds.
I set it as follows:
- color:
- 127
- 0
- 0
id: 0
isthing: 0
name: oil
- color:
- 0
- 0
- 0
id: 1
isthing: 0
name: background
But my losses keep going down but my mIOU stays the same. This question has been bothering me for a long time. Looking forward to your reply.
@plehman2000 did you try with the linear head?
If you still have issues, please take a look at #61 .
tldr: mmsegmentation is actively maintained and you can reach similar performances as this repository with Segmenter.
