CN24 reports incorrect metrics when using test images different in size (WxH) from train images
zygmuntz opened this issue · comments
For example, test F1 etc. are way too high:
INF [ Trainer::Epoch(241) ] Training (Epoch 59, node 0) Square Loss Layer (Weight: 1) lps: 0.0526363
INF [ Trainer::Epoch(243) ] Training (Epoch 59) aggregate lps: 0.0526363
RESULT --- Training - Epoch 59 - F1 : 56.1468% (t=-0.5)
RESULT --- Training - Epoch 59 - ACC: 95.9312%
RESULT --- Training - Epoch 59 - PRE: 52.9711%
RESULT --- Training - Epoch 59 - REC: 59.7277%
RESULT --- Training - Epoch 59 - FPR: 2.41795%
RESULT --- Training - Epoch 59 - FNR: 40.2723%
INF [ NetGraph&, Conv::NetGraph&, Conv::Trainer&, Conv::Trainer&, bool, std::string&)(296) ] Training complete.
> test
INF [ Trainer::Test(117) ] Testing (Epoch 60, node 0) Square Loss Layer (Weight: 1) lps: 0.826714
INF [ Trainer::Test(119) ] Testing (Epoch 60) aggregate lps: 0.826714
RESULT --- Testing - Epoch 60 - F1 : 85.0582% (t=-1)
RESULT --- Testing - Epoch 60 - ACC: 80.2098%
RESULT --- Testing - Epoch 60 - PRE: 99.9539%
RESULT --- Testing - Epoch 60 - REC: 74.0263%
RESULT --- Testing - Epoch 60 - FPR: 0.108757%
RESULT --- Testing - Epoch 60 - FNR: 25.9737%
Trained in hybrid mode.
The term "incorrect metrics" would imply a bug in the implementation of the performance measure. It would be great if you can point us to the piece of source code where you suspect an error. It is also hard for us to reproduce "your results" without having the dataset and especially the way you split it into training and testing subsets.
@erodner Yeah, I know. I'd suggest you try any dataset, any split, with testing images significantly smaller than training images (for example 2x2 or 3x3 smaller). My hunch is you'll be able to reproduce the phenomenon.
Please share at least your network configuration. It may contain parts that are incompatible with hybrid training.
@clrokr Sent by email. Let me know if you need anything else.