Validation Loss while Training
abdksyed opened this issue · comments
Is there a way to get validation loss while training.
I want to fine-tune the model on my dataset, so I am only training for stage-3 for 3000 iterations with batch_size of 8 and 16 num_frames.
But I want to see validation loss or validation IoU on the test set, while training. My concern is maybe the training may overfit, for now I am saving weights after every 50 iterations and trying to see IoU for each weight by running inference.
I think since the model will have memory, it is difficult to do inference and validation metrics during training, but I wanted to know is there any way to do so?
It can definitely be implemented. As you said, it would involve memory updates so the implementation might be a bit hairy.
Is there a way to get validation loss while training.
I want to fine-tune the model on my dataset, so I am only training for stage-3 for 3000 iterations with batch_size of 8 and 16 num_frames.
But I want to see validation loss or validation IoU on the test set, while training. My concern is maybe the training may overfit, for now I am saving weights after every 50 iterations and trying to see IoU for each weight by running inference.
I think since the model will have memory, it is difficult to do inference and validation metrics during training, but I wanted to know is there any way to do so?
Hello, may I ask if you have trained 3000 rounds, num_ How is the effect of setting frames to 16? Have you made any adjustments to the p-value in Celoss? May I ask about the approximate quantity of your dataset? Thank you.