[CLIFF] misc questions for evaluation

Question

[CLIFF] misc questions for evaluation

yohanshin opened this issue a year ago · comments

Soyong Shin commented a year ago

Dear Authors,

Hi, I am trying to reproduce the evaluation table and have a few questions.

I could get similar (not exactly the same though) PA-MPJPE for 3DPW dataset using HR48 backbone. But I want to be confirmed if I am doing it correctly
-- From the previous Issue post, I found that you are using the bbox obtained through projecting 3D keypoints onto image plane. In this case, which keypoints configuration did you use (e.g., COCO17, OpenPose25, H36M ...)
-- I am using the camera focal length as a known value and computed it by averaging the x and y focal lengths. Is it same as what you've done?
Is the HR48 checkpoint uploaded in Google Drive the same as what you got for the Human36M evaluation performance?

Thank you so much for your answer in advance!

zhihaolee · Answer 1 · Fri Apr 28 2023 11:57:59 GMT+0800 (China Standard Time)

1 a). We used the COCO17 keypoint configuration.
1 b). Yes, average of f_x and f_y.
2. No. Please see #138 (comment).

Soyong Shin · Answer 2 · Fri Apr 28 2023 12:05:09 GMT+0800 (China Standard Time)

@zhihaolee

Thank you for your clarifications on the first and the second questions!

For the third one, I have a follow-up question. Does it mean that the checkpoint uploaded on Google Drive will show the MPJPE of 47.6 and PA-MPJPE of 32.7?

zhihaolee · Answer 3 · Fri Apr 28 2023 12:09:59 GMT+0800 (China Standard Time)

Yes, you are right.

Ao Li · Answer 4 · Sun Jul 09 2023 11:16:52 GMT+0800 (China Standard Time)

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

Soyong Shin · Answer 5 · Sun Jul 09 2023 22:04:56 GMT+0800 (China Standard Time)

Hi @RammusLeo , I actually could not get the table for Human3.6M dataset. Sorry for not being so helpful..

Ao Li · Answer 6 · Mon Jul 10 2023 00:24:32 GMT+0800 (China Standard Time)

Thank you all the same for your response! I'll try other methods.

ShamLich · Answer 7 · Sat Jul 29 2023 23:36:02 GMT+0800 (China Standard Time)

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

hi @RammusLeo ,i got the same result as yours,I feel confused

ShamLich · Answer 8 · Sat Jul 29 2023 23:37:28 GMT+0800 (China Standard Time)

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

are these numbers?@RammusLeo
MPJPE: 58.211485621217186
Reconstruction Error: 39.58124800903086