NaN result on: loss, rpn_class_loss and rpn_bbox_loss when Training
sufiaji opened this issue · comments
Hi there,
Thank you for this awesome library.
I manage to download the Nature dataset and try to run the training with below code:
import pixellib
from pixellib.custom_train import instance_custom_training
train_maskrcnn = instance_custom_training()
train_maskrcnn.modelConfig(network_backbone = "resnet50", num_classes= 2, batch_size = 1)
train_maskrcnn.load_pretrained_model("mask_rcnn_coco.h5")
train_maskrcnn.load_dataset("Nature")
train_maskrcnn.train_model(num_epochs = 10, augmentation=True, path_trained_models = "mask_rcnn_models")
However I got NaN loss on 2nd iteration onward as below screencapt:
Please advise to avoid this error.
My environment:
Windows 11 with Nvidia GeForce RTX 3060 laptop and Cuda 10.1
Python 3.8 with Tensorflow 2.2
Using pip, not conda
Thank you in advance