Giters

JDAI-CV / centerX

This repo is implemented based on detectron2 and centernet

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The training curve is strange

iMurphL opened this issue · comments

commented

Hi~ Thank you for your sharing!
I trained three times with different categories in COCO with default configs. The AP curve always dropped suddenly in the middle, then slowly rise. It feels like retraining in the middle. Like this:
image
I checked most of the config params in the yaml files but did not find the reason. Could you tell me which config params made this happened?

commented

I have met this AP curve before， when I set smaller Base_lr，it will be fine.

commented

@iMurphL
Hi, I found an operation in centernet. Maybe this can help.

"For other pytorch version, you can manually open torch/nn/functional.py and find the line with torch.batch_norm and replace the torch.backends.cudnn.enabled with False. We observed slight worse training results without doing so."