Question about evaluation on BADJA
cyrala opened this issue · comments
Hi !
I ran test_on_badja.py and the average PCK-T of pips is 58.7(nvidia 3090), while I founded that in the function of run_pips, the calculation of pck includes the accs of the initial labeled frame(pck =100). But in the function of run_raft, it doesn't include accs of the initial labeled frame, which will lead to the higher pck of pips. I think this processing method is not reasonable. Can you tell me why? And when I calculate pck of pips according to run_raft, I get a result of 53.0(nvidia 3090)
Next, I founded that the test result of the pips' PCK are different on different GPUs(53.0 on nvidia 3090, 51.5 on nvidia 1080), but I don't know why.
The first issue there sounds like a bug. Good catch! I will try to fix this soon, and update the results.
Separate from this, however, it seems like you're getting results a few points lower than the paper (even with the bug included). Is it possible that something else is different in your setup?
For the second issue (differences across GPUs) -- I don't know.
@Rosie-dala I think I fixed the bug in the badja test script.
Check it out: https://github.com/aharley/pips/blob/main/test_on_badja.py#L142-L155