Why I got much worse evaluation results than paper

[Orange Jiang]

I got worse result than paper :

Using Ground Truth this repository provided:
Mean: 29.2925
Median: 29.2096
RMSE: 30.3696
11.25: 2.3061
22.5: 14.5847
30: 57.996
45: 95.9202

Using Ground Truth from silberman:
Mean: 22.4808
Median: 20.1982
RMSE: 26.483
11.25: 29.4958
22.5: 54.2352
30: 67.0443
45: 93.8067

These are much worse than that in paper. But I can not figure out why.
I first use 'imread'(datatype is uint8) to load the predicted picture(0000_pred1.png) and ground truth(0000_gt1.png).

Then I change the RGB->xyz of 0000_pred1.png by pred = pred(:,:,[2 1 3]) and imresize to 224*224. I got pred(0000_pred2.png) and gt(0000_gt2.png).

Then I transfer them to double datatype by NG = double(gt);NP = double(pred); The next steps are normalization and calculate acosin as in eval_pred_sn.m. I found that the picture 0000_pred2.png looks quite different with 0000_gt2.png, which results in bad evaluation results.

Is there anything wrong in my evaluation? Do you know the reason why I got such bad results?

[Yang Yang]

@Juicechen95 Hi do you figure out the reason and get good evaluation results?

I am about to implement the surface normal estimation and found this paper. It seems to have very good results. But the proposed issue make me a little worried about the result in the paper.

Thanks.

[Aayush Bansal]

Perhaps, you are using the wrong files for evaluation. Try out -- https://www.dropbox.com/s/s85qfjq6xq9n63x/marr-revisited.zip?dl=0

The values of normals should be in [-1,1]; convert the uint8 values to double. and divide by 128 and then subtract by 1; and normalize it to 1.

% ----------------------------------------------------------------------------------- %
% ----------------------- SNIPPET OF THE CODE -----------------------------------%

function[nums_e] = eval_pred_sn(cache_dir, cache_list)

    num_images = length(cache_list);
    for i = 1:num_images

            % load the file from cache
            display(['Loading image: ', num2str(i, '%06d'),'/',...
                                        num2str(num_images, '%06d')]);
            pred = load([cache_dir, '/', num2str(cache_list(i), '%06d'),...
                                                     '.mat'],  'sn');
            gtd = load(['./dataset/NYU/GT_Normals/test/nm_',...
                                     num2str(cache_list(i),'%06d') '.mat']);

            %
            NG = cat(3,gtd.nx,gtd.ny,gtd.nz);
            NV = gtd.depthValid;
            %
            NP = pred.rs_sn;
            %normalize both to be sure
            NG = bsxfun(@rdivide,NG,sum(NG.^2,3).^0.5);
            NP = bsxfun(@rdivide,NP,sum(NP.^2,3).^0.5);
            %compute the dot product, and keep on the valid
            DP = sum(NG.*NP,3);
            T = min(1,max(-1,DP));
            pixels{i} = T(find(NV));
    end

    E = acosd(cat(1,pixels{:}));
    nums_e = [mean(E(:)),median(E(:)),mean(E.^2).^0.5,mean(E < 11.25)*100,mean(E < 22.5)*100,mean(E < 30)*100]

end

[Orange Jiang]

Yes, after that I communicate to the author and find that I have used wrong evaluation ground truth. The author provided me the ground truth
and data . With these data, I successfully got the results in the paper.

But it seems the data can not be download now. Unfortunately I do not have the copies...