MIC-DKFZ / medicaldetectiontoolkit

Hi I want to first thank you for posting this toolkit on Github. The whole thing is well-coded!

I learned from the config file that the model selection criteria is AP score, which is awesome! I have used my own dataset and got multiple models (best check points) in exp_dir. Your code generated a file: epoch_ranking.npy, which contains ranked epoch numbers. I noticed there are multiple precision-recall graphs in plots folder, however, precision-recall graphs don't have AP score on them.

If you don't mind me asking, how can I find the AP score of each of these epoch numbers ranked in epoch_ranking.npy?

Thank you very much!
-Bo

Hi, thanks for your feedback.

The scores are not written out in epoch_ranking.py, but you could change that around here:

medicaldetectiontoolkit/utils/exp_utils.py

Line 150 in 8a2218f

    
           non_nan_scores = np.mean(np.array([[0 if ii is None else ii for ii in monitor_metrics['val'][sc]] for sc in self.cf.model_selection_criteria]), 0)

Otherwise, did you see the monitoring plot, that displays AP values for all classes over epochs?

Hi, thanks for your feedback.

The scores are not written out in epoch_ranking.py, but you could change that around here:

medicaldetectiontoolkit/utils/exp_utils.py

Line 150 in 8a2218f

non_nan_scores = np.mean(np.array([[0 if ii is None else ii for ii in monitor_metrics['val'][sc]] for sc in self.cf.model_selection_criteria]), 0)

Otherwise, did you see the monitoring plot, that displays AP values for all classes over epochs?

Hi pfjaeger, I want to thank you for instant reply! I now know how to extract AP score of the best models. Due to WiFi issue, I accidentally clicked 'close'.

I can see 'monitor_0_0.png' of which the x axis is 'epochs' and y axis is 'loss / metrices'. Is 'loss / metrices' in this graph AP?

Thank you. -Bo

There should be a legend to the plot as in https://github.com/MIC-DKFZ/medicaldetectiontoolkit/blob/master/assets/loss_monitoring.png ?

Yes that's right. Thank you very much for getting back to me and help me solve this problem. I appreciate it! -Bo

How to find AP score of the best trained model?