Metrics
chx-Github opened this issue · comments
Hi! @kbrodt Thanks for sharing this great code！
Where are the codes of the evaluation metrics (for example: ADE, FDE, minADE, minFDE and so on)? Or where can I find it?
Looking forward to your reply！
Hi!
There is no code of metrics. You may submit your predictions at https://waymo.com/open/challenges/2021/motion-prediction/
I see, thanks for your reply!
@kbrodt Sorry to bother you again!
When submitting predictions at https://waymo.com/open/challenges/2021/motion-prediction/, which compressed files should be submitted to get the correct evaluation results? Is it the output result of the submit.py file? What format file should the output result of this submit.py file be saved in? Or do I need to submit other files?
Looking forward to your reply!
Yes, the only file to sumbit is an output of submit.py (flag --save).
@kbrodt Thank you very much for your last answer, but there is another problem that needs your help.
When I uploaded the output compressed file of my submit.py in the Validation Set at https://waymo.com/open/challenges/2021/motion-prediction/, I got the following error:
INVALID_ARGUMENT: Not enough scenario predictions in submission : 19244
Is the error caused by the incomplete download of the training set and verification set? Because I only downloaded 1/10 of the training set and 2/5 of the verification set. Or are there other unknown errors?
Looking forward to your reply!
I guess you need full validation dataset for the submition through site https://waymo.com/open/challenges/2021/motion-prediction/
OK, thank you very much for your suggestion. I'll continue to try the full validation dataset.
@kbrodt Sorry to bother you again, there is a puzzling problem. When I run the train.py file, the loss rate of its output is as follows:
`C:\Users\Administrator.DESKTOP-HRTI73G\anaconda3\envs\DL\python.exe F:/waymo-motion-prediction-2021-2/train.py --train-data ./train --dev-data ./dev --save ./xception71 --model xception71 --img-res 224 --in-channels 25 --time-limit 80 --n-traj 6 --lr 0.001 --batch-size 48 --n-epochs 10
2022-05-05 22:47:49.278918: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
loss: 1.42e+04 avg: 2.1e+04 last_lr: 0.001: 100%|██████████| 10390/10390 [2:53:22<00:00, 1.00s/it]`
It's surprising that the loss rate is so great? e+04?
I read the formula of loss rate in your report and the corresponding code, I feel it is correct, but why is the output of loss rate so large? Is there something wrong somewhere else?
I'ts ok, the loss of untrained model is high for the few first epochs. In report we cut first epochs for better layout.