Low accuracy in pretrained model on Market-1501

[snowaner]

Hello there,

I ran the "Prepare the Pretrained Models" with backbone ResNet50 on 4GPUs with batch_size 64. All settings are unchanged and the command is "sh scripts/pretrain.sh dukemtmc market1501 resnet50 1". However, I only get 40.7%/59.5% (mAP/Rank-1) on source dataset (DukeMTMC-reID), and 19.1%/42.6% on target dataset (Market-1501). Any idea why it might be happening, since the reported performance on target data is 31.8%/61.9%.

Some phenomenons

1. the triplet loss is about 0.5 in average when training is over
2. the precision is about 100% when training is over

Besides, I have some questions about settings

1. the "margin" is set 0.0 in scripts/pretrain.sh, while the previous UDA methods use a "margin" of 0.5 (Sec 3.1 in paper)
2. "iters" is set 100 in scripts/pretrain.sh.

Thanks a lot!

[Yixiao Ge]

Hi,

I have just re-trained my code again to find if there exist any issues in the pre-training stage (./scripts/pretrain.sh dukemtmc market1501 resnet50 1). And the results are satisfactory, i.e. on dukemtmc

Mean AP: 69.5%
CMC Scores:
  top-1          85.0%
  top-5          92.0%
  top-10         94.3%

and on market1501

Mean AP: 29.6%
CMC Scores:
  top-1          58.2%
  top-5          73.5%
  top-10         79.3%

Although 29.6% is not as high as 31.8% (as reported), I don't think it would affect the performance of the following MMT stage. And the reasons for such a slight drop may be due to the fact that (1) randomness of training and (2) the decreased iter number (200->100). The reason for the decreased iter number refers to the below.

As for the phenomenons you provided, I think the value of triplet loss is abnormal. The triplet loss is about 0.024 in my experiments, e.g.

Epoch: [79][100/100]    Time 0.253 (0.277)      Data 0.000 (0.014)      Loss_ce 1.086 (1.076)   Loss_tr 0.037 (0.024)   Prec 100.00% (99.98%)

Maybe you need to check the convergence of the network. Did you modify anywhere of the code?

As for your questions,

1. margin: in the section 3.1 of the paper, the margin=0.5 is for the conventional triplet loss. And here in my pre-training code, we use hard-version softmax-triplet loss (Eq. (6) in the paper), so margin=0 is better.
2. iters: I use iters=200 in the original paper's experiments. And later, I found that iters=100 could achieve similar pre-training performance but with much faster speed than iters=200. So I modify the scripts.

[snowaner]

Hi, thank you for your reply.

I have finally found the problem and achieve 30.1%/58.3% on target dataset (Market1501).
As you suggest, the convergence of the network went wrong due to the reset of the initialization.

For some reasons, I had to load the pretrained backbone of ResNet50 from the local (self-defined path). Therefore, I set the parameter "pretrained" as False in Class "models", and manually loaded the pretrained backbone of ResNet50.

I revised "mmt/models/resnet.py" like this for loading the pretrained backbone of ResNet50 from the local
if depth not in ResNet.__factory:
raise KeyError("Unsupported depth:", depth)
resnet = ResNet.__factorydepth
resnet.load_state_dict(torch.load(LOCAL_PATH+ 'resnet50-19c8e357.pth'))

However, when the "pretrained" is set False, it will lead to the reset of the initialization
if not pretrained:
self.reset_params()

Thus, the network will be randomly initialized and not loaded from the pretrained backbone, and finally lead to the poor convergence.

Thanks again for the reply of my questions, it is really helpful.