Question about pretrained weights performance

Question

Question about pretrained weights performance

taeyeopl opened this issue 3 years ago · comments

I used the provided checkpoints vgg16_ycb_video_epoch_16.checkpoint.pth to check the YCB dataset(ycb_video_keyframe) performance. I got the below results. I have some questions related to the results.

Q1. Do pretrained weights show same pretrained weights of the paper PoseCNN and DexYCB?? If not, can you share your results based on all the pretrained results of all the datasets(ycb_video, dex_ycb, ycb_object)?? I only observed that the YCB dataset performance except other datasets (dex_ycb_s0~3, ycb_object).

ADD: 0.537 | [Provided checkpoints 0.536727]
ADD-S: 0.759 | [Provided checkpoints 0.743643]
ADD refined: 0.793 | [Provided checkpoints 0.075520]
ADD-S refined: 0.993 | [Provided checkpoints 0.119003]

Q2. I observed that refined performance was quite lower than reported. I felt it is wired. Can you explain why??

time ./tools/test_net.py --gpu 0 \
  --network posecnn \
  --pretrained data/checkpoints/ycb_video/vgg16_ycb_video_epoch_16.checkpoint.pth \
  --dataset ycb_video_keyframe \
  --cfg experiments/cfgs/ycb_video.yml
  
==================My Performance(ALL)======================
ADD: 0.536727
ADD-S: 0.743643
ADD refined: 0.075520
ADD-S refined: 0.119003
==================ADD======================
all: 0.536727
002_master_chef_can: 0.615168
003_cracker_box: 0.617424
004_sugar_box: 0.532164
005_tomato_soup_can: 0.708913
006_mustard_bottle: 0.797891
007_tuna_fish_can: 0.674004
008_pudding_box: 0.479661
009_gelatin_box: 0.770318
010_potted_meat_can: 0.688317
011_banana: 0.708651
019_pitcher_base: 0.723790
021_bleach_cleanser: 0.473929
024_bowl: 0.062281
025_mug: 0.534518
035_power_drill: 0.547650
036_wood_block: 0.023127
037_scissors: 0.465419
040_large_marker: 0.591836
051_large_clamp: 0.109451
052_extra_large_clamp: 0.032313
061_foam_brick: 0.701915
0.615168
0.617424
0.532164
0.708913
0.797891
0.674004
0.479661
0.770318
0.688317
0.708651
0.723790
0.473929
0.062281
0.534518
0.547650
0.023127
0.465419
0.591836
0.109451
0.032313
0.701915
0.536727
ycb_video
===========================================
==================ADD-S====================
all: 0.743643
002_master_chef_can: 0.877846
003_cracker_box: 0.803593
004_sugar_box: 0.741124
005_tomato_soup_can: 0.838784
006_mustard_bottle: 0.900914
007_tuna_fish_can: 0.874633
008_pudding_box: 0.700458
009_gelatin_box: 0.885764
010_potted_meat_can: 0.859981
011_banana: 0.868941
019_pitcher_base: 0.862067
021_bleach_cleanser: 0.654374
024_bowl: 0.730333
025_mug: 0.761190
035_power_drill: 0.748332
036_wood_block: 0.257070
037_scissors: 0.631964
040_large_marker: 0.718532
051_large_clamp: 0.412216
052_extra_large_clamp: 0.308376
061_foam_brick: 0.851005
0.877846
0.803593
0.741124
0.838784
0.900914
0.874633
0.700458
0.885764
0.859981
0.868941
0.862067
0.654374
0.730333
0.761190
0.748332
0.257070
0.631964
0.718532
0.412216
0.308376
0.851005
0.743643
ycb_video
===========================================
==================ADD refined======================
all: 0.075520
002_master_chef_can: 0.123327
003_cracker_box: 0.046552
004_sugar_box: 0.077753
005_tomato_soup_can: 0.065748
006_mustard_bottle: 0.079828
007_tuna_fish_can: 0.135382
008_pudding_box: 0.047530
009_gelatin_box: 0.047532
010_potted_meat_can: 0.088641
011_banana: 0.081146
019_pitcher_base: 0.097756
021_bleach_cleanser: 0.041396
024_bowl: 0.008421
025_mug: 0.172956
035_power_drill: 0.085137
036_wood_block: 0.040056
037_scissors: 0.046730
040_large_marker: 0.060406
051_large_clamp: 0.048245
052_extra_large_clamp: 0.021109
061_foam_brick: 0.058939
0.123327
0.046552
0.077753
0.065748
0.079828
0.135382
0.047530
0.047532
0.088641
0.081146
0.097756
0.041396
0.008421
0.172956
0.085137
0.040056
0.046730
0.060406
0.048245
0.021109
0.058939
0.075520
ycb_video
===========================================
==================ADD-S refined====================
all: 0.119003
002_master_chef_can: 0.173956
003_cracker_box: 0.081636
004_sugar_box: 0.107987
005_tomato_soup_can: 0.092742
006_mustard_bottle: 0.108868
007_tuna_fish_can: 0.191927
008_pudding_box: 0.069155
009_gelatin_box: 0.058518
010_potted_meat_can: 0.130085
011_banana: 0.119868
019_pitcher_base: 0.124502
021_bleach_cleanser: 0.060756
024_bowl: 0.074100
025_mug: 0.189300
035_power_drill: 0.123411
036_wood_block: 0.179038
037_scissors: 0.056125
040_large_marker: 0.081108
051_large_clamp: 0.112593
052_extra_large_clamp: 0.181822
061_foam_brick: 0.088207
0.173956
0.081636
0.107987
0.092742
0.108868
0.191927
0.069155
0.058518
0.130085
0.119868
0.124502
0.060756
0.074100
0.189300
0.123411
0.179038
0.056125
0.081108
0.112593
0.181822
0.088207
0.119003
ycb_video
===========================================

Process finished with exit code 0

Jiangran@PKU · Answer 1 · Sat Jul 23 2022 11:57:19 GMT+0800 (China Standard Time)

I used the provided checkpoints vgg16_ycb_video_epoch_16.checkpoint.pth to check the YCB dataset(ycb_video_keyframe) performance. I got the below results. I have some questions related to the results.

Q1. Do pretrained weights show same pretrained weights of the paper PoseCNN and DexYCB?? If not, can you share your results based on all the pretrained results of all the datasets(ycb_video, dex_ycb, ycb_object)?? I only observed that the YCB dataset performance except other datasets (dex_ycb_s0~3, ycb_object).

* ADD: 0.537 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.536727]

* ADD-S: 0.759 |  [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.743643]

* ADD refined: 0.793 | [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.075520]

* ADD-S refined: 0.993 |  [[Provided checkpoints](https://drive.google.com/file/d/1-ECAkkTRfa1jJ9YBTzf04wxCGw6-m5d4/view) 0.119003]

Q2. I observed that refined performance was quite lower than reported. I felt it is wired. Can you explain why??

time ./tools/test_net.py --gpu 0 \
  --network posecnn \
  --pretrained data/checkpoints/ycb_video/vgg16_ycb_video_epoch_16.checkpoint.pth \
  --dataset ycb_video_keyframe \
  --cfg experiments/cfgs/ycb_video.yml
  
==================My Performance(ALL)======================
ADD: 0.536727
ADD-S: 0.743643
ADD refined: 0.075520
ADD-S refined: 0.119003
==================ADD======================
all: 0.536727
002_master_chef_can: 0.615168
003_cracker_box: 0.617424
004_sugar_box: 0.532164
005_tomato_soup_can: 0.708913
006_mustard_bottle: 0.797891
007_tuna_fish_can: 0.674004
008_pudding_box: 0.479661
009_gelatin_box: 0.770318
010_potted_meat_can: 0.688317
011_banana: 0.708651
019_pitcher_base: 0.723790
021_bleach_cleanser: 0.473929
024_bowl: 0.062281
025_mug: 0.534518
035_power_drill: 0.547650
036_wood_block: 0.023127
037_scissors: 0.465419
040_large_marker: 0.591836
051_large_clamp: 0.109451
052_extra_large_clamp: 0.032313
061_foam_brick: 0.701915
0.615168
0.617424
0.532164
0.708913
0.797891
0.674004
0.479661
0.770318
0.688317
0.708651
0.723790
0.473929
0.062281
0.534518
0.547650
0.023127
0.465419
0.591836
0.109451
0.032313
0.701915
0.536727
ycb_video
===========================================
==================ADD-S====================
all: 0.743643
002_master_chef_can: 0.877846
003_cracker_box: 0.803593
004_sugar_box: 0.741124
005_tomato_soup_can: 0.838784
006_mustard_bottle: 0.900914
007_tuna_fish_can: 0.874633
008_pudding_box: 0.700458
009_gelatin_box: 0.885764
010_potted_meat_can: 0.859981
011_banana: 0.868941
019_pitcher_base: 0.862067
021_bleach_cleanser: 0.654374
024_bowl: 0.730333
025_mug: 0.761190
035_power_drill: 0.748332
036_wood_block: 0.257070
037_scissors: 0.631964
040_large_marker: 0.718532
051_large_clamp: 0.412216
052_extra_large_clamp: 0.308376
061_foam_brick: 0.851005
0.877846
0.803593
0.741124
0.838784
0.900914
0.874633
0.700458
0.885764
0.859981
0.868941
0.862067
0.654374
0.730333
0.761190
0.748332
0.257070
0.631964
0.718532
0.412216
0.308376
0.851005
0.743643
ycb_video
===========================================
==================ADD refined======================
all: 0.075520
002_master_chef_can: 0.123327
003_cracker_box: 0.046552
004_sugar_box: 0.077753
005_tomato_soup_can: 0.065748
006_mustard_bottle: 0.079828
007_tuna_fish_can: 0.135382
008_pudding_box: 0.047530
009_gelatin_box: 0.047532
010_potted_meat_can: 0.088641
011_banana: 0.081146
019_pitcher_base: 0.097756
021_bleach_cleanser: 0.041396
024_bowl: 0.008421
025_mug: 0.172956
035_power_drill: 0.085137
036_wood_block: 0.040056
037_scissors: 0.046730
040_large_marker: 0.060406
051_large_clamp: 0.048245
052_extra_large_clamp: 0.021109
061_foam_brick: 0.058939
0.123327
0.046552
0.077753
0.065748
0.079828
0.135382
0.047530
0.047532
0.088641
0.081146
0.097756
0.041396
0.008421
0.172956
0.085137
0.040056
0.046730
0.060406
0.048245
0.021109
0.058939
0.075520
ycb_video
===========================================
==================ADD-S refined====================
all: 0.119003
002_master_chef_can: 0.173956
003_cracker_box: 0.081636
004_sugar_box: 0.107987
005_tomato_soup_can: 0.092742
006_mustard_bottle: 0.108868
007_tuna_fish_can: 0.191927
008_pudding_box: 0.069155
009_gelatin_box: 0.058518
010_potted_meat_can: 0.130085
011_banana: 0.119868
019_pitcher_base: 0.124502
021_bleach_cleanser: 0.060756
024_bowl: 0.074100
025_mug: 0.189300
035_power_drill: 0.123411
036_wood_block: 0.179038
037_scissors: 0.056125
040_large_marker: 0.081108
051_large_clamp: 0.112593
052_extra_large_clamp: 0.181822
061_foam_brick: 0.088207
0.173956
0.081636
0.107987
0.092742
0.108868
0.191927
0.069155
0.058518
0.130085
0.119868
0.124502
0.060756
0.074100
0.189300
0.123411
0.179038
0.056125
0.081108
0.112593
0.181822
0.088207
0.119003
ycb_video
===========================================

Process finished with exit code 0

Hello Lee, I find that you have test the checkpoint, so i'd like to seek some help from you. My own checkpoint(obtained from training code) get worse performance(ADD = 51,ADD-S = 73 after refinement) than paper report(ADD = 79, ADD-S = 93 after refinement). I'd like to ask you that whether you meet the same situation with me. Thank you!
@yuxng I'll be very appreciate if yu xiang can offer some help