Giters

valeoai / obsnet

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

About results on the Segment me if you can datasets

haoz19 opened this issue · comments

commented

Hello ,

Thanks for your great work and your work has inspired me a lot.

I see the result is pretty good on Segment me if you can dataset, but when I follow your instruction and download the pre-trained model for Segment me if you can dataset, I got results like these: FPR@TPR95: 0.617; AUPRC: 0.569; AUROC: 0.862, which is much different than which is on the leaderboard: AUPRC: 0.754, FPR@TPR95: 0.267. (But I did the annotation of SMIYC myself, so the result may be different.) Is that possible that you can send me the anomaly score (.hdf5) for the Segment me if you can dataset, that you send to the organizer of SMIYC, so I can check if my result is right?

It'll be a lot of help for my current work!

Many Thanks

commented

Hi @haoz19,
What was the performance on Segment me if you can Road anomaly validation dataset? I am getting very poor results ?

commented

I got FPR@TPR95: 0.402; AP: 0.727; AUROC: 0.919 in the SMIYC Road anomaly validation set (10 images).

commented

Did you get similar results?

commented

Unfortunately no. Did you use evaluation.py or inference.py?

commented
commented

Okay I see. I am getting
AUROC score: 0.7873951347082428
AUPRC score: 0.40464778006347774
FPR@TPR95: 0.5757097820937631
which is lot worse.
One question did you interpolate the final anomaly scores as it comes with a fixed size ?

commented
commented

Thank you so much !! Also, the results get worse when I test it on road obstacle
AUROC score: 0.8077338939302358
AUPRC score: 0.12090705488023645
FPR@TPR95: 0.5609430880181637

commented
commented

No. Because I think the authors released the pre-trained models for SMIYC benchmark. So I stick to Road Obstacle and Road Anomalies dataset. But I will give a try !!

commented

But did you see a similar drop in Accuracy while testing it on road obstacle dataset ?

commented
commented

Hello,

You can find the (.hdf5) file submitted on the leaderboard in the following google drive link: https://drive.google.com/drive/folders/1yDT5dJsWoxRapROTRRUVPqXF0qTId85E?usp=sharing

You should be able to reproduce the results by using the inference.py file (for example):
python inference.py --data "CityScapes" --model "road_anomaly" --img_folder "./data/dataset_AnomalyTrack/images/" --segnet_file "./ckpt/WideResnet_DeepLabv3plus_CityScapes.pth" --obsnet_file "./ckpt/ObsNet_CityScapes.pth"

The pre-trained models are available on the README.md or here:
https://drive.google.com/drive/folders/1Xxrl0JPU1KNdrZSFTBG1S2be5eu0XjPS?usp=share_link

We did not test our method on the obstacle track of SegmentMeIfYouCan.

Hoping that it helps 🙂

Victor

commented

I just meet with the author and now I can get the result. I used to reshape the image, and when I use the code without any change, I can get the result!