Why I got much worse evaluation results than paper
Juicechen95 opened this issue · comments
I got worse result than paper :
Using Ground Truth this repository provided:
Mean: 29.2925
Median: 29.2096
RMSE: 30.3696
11.25: 2.3061
22.5: 14.5847
30: 57.996
45: 95.9202
Using Ground Truth from silberman:
Mean: 22.4808
Median: 20.1982
RMSE: 26.483
11.25: 29.4958
22.5: 54.2352
30: 67.0443
45: 93.8067
These are much worse than that in paper. But I can not figure out why.
I first use 'imread'(datatype is uint8) to load the predicted picture(0000_pred1.png) and ground truth(0000_gt1.png).
Then I change the RGB->xyz of 0000_pred1.png by pred = pred(:,:,[2 1 3]) and imresize to 224*224. I got pred(0000_pred2.png) and gt(0000_gt2.png).
Then I transfer them to double datatype by NG = double(gt);NP = double(pred); The next steps are normalization and calculate acosin as in eval_pred_sn.m. I found that the picture 0000_pred2.png looks quite different with 0000_gt2.png, which results in bad evaluation results.
Is there anything wrong in my evaluation? Do you know the reason why I got such bad results?
@Juicechen95 Hi do you figure out the reason and get good evaluation results?
I am about to implement the surface normal estimation and found this paper. It seems to have very good results. But the proposed issue make me a little worried about the result in the paper.
Thanks.
Perhaps, you are using the wrong files for evaluation. Try out -- https://www.dropbox.com/s/s85qfjq6xq9n63x/marr-revisited.zip?dl=0
The values of normals should be in [-1,1]; convert the uint8 values to double. and divide by 128 and then subtract by 1; and normalize it to 1.
% ----------------------------------------------------------------------------------- %
% ----------------------- SNIPPET OF THE CODE -----------------------------------%
function[nums_e] = eval_pred_sn(cache_dir, cache_list)
num_images = length(cache_list);
for i = 1:num_images
% load the file from cache
display(['Loading image: ', num2str(i, '%06d'),'/',...
num2str(num_images, '%06d')]);
pred = load([cache_dir, '/', num2str(cache_list(i), '%06d'),...
'.mat'], 'sn');
gtd = load(['./dataset/NYU/GT_Normals/test/nm_',...
num2str(cache_list(i),'%06d') '.mat']);
%
NG = cat(3,gtd.nx,gtd.ny,gtd.nz);
NV = gtd.depthValid;
%
NP = pred.rs_sn;
%normalize both to be sure
NG = bsxfun(@rdivide,NG,sum(NG.^2,3).^0.5);
NP = bsxfun(@rdivide,NP,sum(NP.^2,3).^0.5);
%compute the dot product, and keep on the valid
DP = sum(NG.*NP,3);
T = min(1,max(-1,DP));
pixels{i} = T(find(NV));
end
E = acosd(cat(1,pixels{:}));
nums_e = [mean(E(:)),median(E(:)),mean(E.^2).^0.5,mean(E < 11.25)*100,mean(E < 22.5)*100,mean(E < 30)*100]
end
Yes, after that I communicate to the author and find that I have used wrong evaluation ground truth. The author provided me the ground truth
and data . With these data, I successfully got the results in the paper.
But it seems the data can not be download now. Unfortunately I do not have the copies...