Could you share the testing code?
BlingHe opened this issue · comments
Hi,
I'm trying to replicate your reported results. Using “run_infer.py” with the default settings (ensemble_size=3 and denoise_steps=10) for surface normal estimation, my scores for Mean Angular Error and Accuracy (11.25) are lower than those in your paper.
Could you provide more details on your testing setup or share the relevant testing code? It would help me better understand your findings. Thanks in advance!
Below are the results obtained by me compared to those you reported.
Hi, thanks for the ask. In testing, we set the denoising step as 50 and ensemble size as 10. The default settings (ensemble_size=3 and denoise_steps=10) are for quick inference on in-the-wild images. Besides, as explained in Fig. S2, we found that the GT normals are very noisy (especially on NYU V2), thus we did some filtering on normals. The release of the evaluation script as well as the training script is still on the schedule.
Ok, I see! Thank you so much for your reply.
Besides, I think an interesting direction for normal estimation is like line2normal map. We should not put too much attention on traditional estimation benchmarks, but more on the in-the-wild applications. Line2normal map displays good visual effect, but its normal map is conditoned on lines, thus the accuracy must be low. However, the lighting effect achieved on normal map is impressive, thus the user will not basically care about whether the normal is accurate or not.
Exactly. However, I want to understand how estimated surface normals can benefit downstream applications. The question remains open: Which is more crucial for generative applications (e.g., 3D reconstruction)—accurate normals or depths, or are the additional normal and depth conditions more critical? Evaluating the estimated normals and depths using traditional benchmarks may provide some inspiration.
This is my opinion, and your work is very interesting and inspiring.