ayoolaolafenwa / PixelLib

Visit PixelLib's official documentation https://pixellib.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

NaN result on: loss, rpn_class_loss and rpn_bbox_loss when Training

sufiaji opened this issue · comments

Hi there,

Thank you for this awesome library.
I manage to download the Nature dataset and try to run the training with below code:

import pixellib
from pixellib.custom_train import instance_custom_training

train_maskrcnn = instance_custom_training()
train_maskrcnn.modelConfig(network_backbone = "resnet50", num_classes= 2, batch_size = 1)
train_maskrcnn.load_pretrained_model("mask_rcnn_coco.h5")
train_maskrcnn.load_dataset("Nature")
train_maskrcnn.train_model(num_epochs = 10, augmentation=True, path_trained_models = "mask_rcnn_models")

However I got NaN loss on 2nd iteration onward as below screencapt:

image

Please advise to avoid this error.

My environment:
Windows 11 with Nvidia GeForce RTX 3060 laptop and Cuda 10.1
Python 3.8 with Tensorflow 2.2
Using pip, not conda

Thank you in advance