Evaluating PSMNet on KITTI VO with Pretrained KITTI-2012 model delivers biased and skewed disparity results

[Jonas]

I have looked at the KITTI Stereo 2012 dataset, which features parts of the KITTI VO dataset. I conclude that they are part of the same set. This means that I can just use the pretrained 2012 model.
Now, when I evaluate it on sequence 04 of the VO dataset, my disparity values seem to be off compared to ELAS and other stereo algorithms:

Also I tried out the other pretrained models. I compared the velodyne projected to the color image. Also, I altered/improved the velodyne to camera calibration, which is somewhat off for the VO dataset.

COMMAND USED: python submission.py --KITTI 2012 --datapath C:\KittiOdometry\data_odometry_color\dataset\sequences\04\ --loadmodel pretrained_model_KITTI2012.tar
Note that I altered the KITTI_submission_loader2012.py to work with KITTI VO instead.

Here's my resulting image of the first pair:

Here are the two input pictures. It would be really nice if somebody would post their disparity results of these two images with the pretrained KITTI 2012 model and hourglass. Then we could have a comparison and identify the source of the error faster.

[Jonas]

@JiaRenChang Any advice?

[Jia-Ren Chang]

@WASCHMASCHINE
Hi, I would like to know that the sequence 04 of the VO dataset performs badly?

Here is my testing result, I think it is same as your testing result.

[Jonas]

@JiaRenChang I think I see no difference between our images.

I decided to evaluate the KITTI2015 model on the KITTI2015 training. It still looks quite off, maybe my evaluation code is wrong? But other disparity results look OK. Please disregard the oversampling.

My comparison code looks like this:
Importer

newFrame10.groundTruthDisparityImage = cv::imread(_filepath + "\\data_scene_flow\\training\\disp_noc_0\\" + scenePaddingString + "_10.png", CV_LOAD_IMAGE_UNCHANGED);
newFrame10.groundTruthDisparityImage.convertTo(newFrame10.groundTruthDisparityImage, CV_32FC1, 1.0f / 256.0f); //UINT16 -> FLOAT32
newFrame10.precomputedDisparityImage = cv::imread(_filepath + "\\precomputed_psmnet_disparities\\" + scenePaddingString + "_10.png", CV_LOAD_IMAGE_UNCHANGED);
newFrame10.precomputedDisparityImage.convertTo(newFrame10.precomputedDisparityImage, CV_32FC1, 1.0f / 256.0f); //UINT16 -> FLOAT32

Evaluation

float gtDisparity = groundTruthDisparity.at<float>(imageHeight - 1 - v, u);
float reconstructedDisparity = stereoDisparity.at<float>(imageHeight - 1 - v, u);
if (gtDisparity <= 0) continue; // No ground truth available
if (!isnormal(reconstructedDisparity) || reconstructedDisparity <= 0.0f ) continue;
float deltaDisparity = reconstructedDisparity - gtDisparity;

Have you computed the error on the training dataset?

[Jonas]

I ran the devkit sample from KITTI 2015 and implemented the accuracy measure in my code again.
Using the NOC GT I compared the disparity error: They look the same and pretty good (example frame 154: 1.35% accuracy). But I use the same error for my distribution. I conclude that the results of PSMNet are biased, but still very good in terms of KITTI accuracy. Subtracting -1.25 px per pixel from the reconstructed disparities decreased the error even further! @JiaRenChang I would suggest that you may try out evaluating PSMNet with a bias correction.

Avg. Accuracy on Kitti 2015 Training (200 frames, quick hack)
0.91 % Vanilla
0.67 % With -1.25 px on disparities

[Jia-Ren Chang]

@WASCHMASCHINE

Wow, many thanks for your detailed evaluation!
It is interesting that why PSMNet is biased for 1.25 px.
I will try to figure out it.
Thanks again!!

[Sarah]

@WASCHMASCHINE For KITTI2015, if I trained PSMNet (finetuned from sceneflow dataset) on 160 frames and test on the rest 40 frames, -1.25 px on disparities cannot get a better result in my experiment.

[Jonas]

@zhFuECL Why don't you compute a disparity error histogram like I did?

I used the whole training set of Kitti 2015 to compute the histogram, looked up the bias, and reevaluated it on the training set. I used the pretrained model from the author without modifications for disparities.

[Jonas]

@JiaRenChang Any news on the topic?
I tested the pretrained_model of 2015 with the devkit matlab scripts and it looked similar but not the same as your result image on the KITTI website. I am not sure what's going on -- do I have to finetune the pretrained model?

[tv12345]

Hello @WASCHMASCHINE did you use the standard 1242x375 images from the KITTI-2015 testing image folder? or did you scale them down to a lower resolution?
I am trying to get the algorithm working with the pretrained model provided to calculate depth on Jeston TX2.