How to only keep final (best) weights
Dilshad737 opened this issue · comments
I run the training on my custom dataset, I annotated the dataset using labelme tool. All things go fine. But I have a concern. I trained the model for 20 epochs. I got following 12 different .h5 files.
mask_rcnn_model.001-0.507531.h5 mask_rcnn_model.002-0.196400.h5 mask_rcnn_model.003-0.142704.h5 mask_rcnn_model.004-0.115689.h5 mask_rcnn_model.005-0.105847.h5 mask_rcnn_model.006-0.074317.h5 mask_rcnn_model.008-0.054073.h5 mask_rcnn_model.009-0.050936.h5 mask_rcnn_model.012-0.044279.h5 mask_rcnn_model.013-0.041470.h5 mask_rcnn_model.014-0.041182.h5 mask_rcnn_model.016-0.036677.h5
I should get 20 files but I got 12 why?
These 12 consumes a lot of memory in my google colab. Is there a way to remove previous .h5 files on the go and just keep the best weights. Only 1 weight at the end of training with best weights.
@ayoolaolafenwa Any idea?
Hi, maybe this can help you matterport/Mask_RCNN#747 (comment)
@Dilshad737 Models are saved based on the decrease in the validation losses e.g during training if an epoch with a low validation loss is saved it will not save any other weight until it reaches another epoch with a lower loss than the saved one.
Check the custom training tutorial
@pablobots @ayoolaolafenwa Thanks for the assistance, I got my issue resolved.