WP 3.1: Better SFM
ErlerPhilipp opened this issue · comments
"Genauere Recherche zusammen mit Stefan, um die beste Variante von SFM für die in den Tests gefundenen Bedingungen auszuwählen.
Implementierungsaufwand abwägen.
Implementieren, evtl. in C++, sonst wahrscheinlich in Python."
Discuss this with Attila from the VRVis. One option is AliceVision.
Another, more experimental, solution is to try a NeRF with surface extraction.
AliceVision was not that good in my tests. Worse reconstructions, similar speed as COLMAP.
The way forward seems to be with NeRFs. The most promising being:
- NVDiffRec: https://github.com/NVlabs/nvdiffrec
- Nerfstudio (Nerfacto): https://github.com/nerfstudio-project/nerfstudio
i would prefer Nerfstudio because it has a nicer license
Nerfstudio just released a new version with NeuS and NeuSFacto model based on this paper: https://arxiv.org/pdf/2106.10689.pdf
Could be worth a try.
interesting paper by a colleague: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
maybe wait and see if it will be included in nerfstudio
another interesting work that could replace COLMAP: https://openaccess.thecvf.com/content/CVPR2023/papers/Li_Neuralangelo_High-Fidelity_Neural_Surface_Reconstruction_CVPR_2023_paper.pdf
and another: https://liuyuan-pal.github.io/NeRO/
the experiments so far were not convincing. all nerf methods have either too strict requirements to the inputs (accurate camera poses, images from all sides etc.) or worse quality than COLMAP (although the paper results look better)
had a longer discussion with a visitor... NeuS and derivates are probably the way to go. we care about the geometry, which typical NeRFs can't represent properly.
note that we need a version without segmentation masks that also works on unbounded scenes. apparently, they all depend on COLMAP for camera poses.
NeuS (rather slow): https://github.com/Totoro97/NeuS
NeuDA (slow?): https://github.com/3D-FRONT-FUTURE/NeuDA
NeuS++ (faster, check branch for unmasked): https://github.com/19reborn/NeuS2/tree/neuspp
Instant Neural Surface Reconstruction(fast): https://github.com/bennyguo/instant-nsr-pl
NeRO (slow?, no unbounded scenes?): https://github.com/liuyuan-pal/NeRO
Neuralangelo (need license): https://research.nvidia.com/labs/dir/neuralangelo/
SDFStudio (like NeRFStudio but for reconstruction, check license for angelo): https://github.com/autonomousvision/sdfstudio
most interesting seem SDFStudio and NeuS++
@ErlerPhilipp Hi Philipp. I have recently released a derivative version of Neus and Neuralangelo called Instant-angelo. This new project has a friendly open-source license, higher efficiency, and is fully compatible with Colmap. Additionally, while our current reconstruction pipeline does not use masking, it focuses on reconstructing just the central area within the camera poses' field of view. I hope this new project will be useful to the research community. We provide some visualization of current results. Please let me know if you have any other questions!
Hi @hugoycj ! Thanks, your work looks indeed very interesting.
I have several questions:
- There is no official publication, right? Do you plan to publish it?
- Are the implementation details described or documented somewhere or should I just look at the code?
- Can I download some results as textured meshes? I see only GIFs in your repo.
- In case you have the time, I would really appreciate if you could run this example with your method: https://drive.google.com/drive/folders/1AgSzlI884Hezh4ebPBe5dITLAI8fX4He?usp=sharing
Apologies for the delayed response as I am currently occupied with the deadline for a conference.
Instant-Angelo is still in the developmental phase. Its functionality heavily relies on the results obtained through sfm (structure from motion) and may encounter issues if the sfm points are noisy or sparse. Our main objective for developing Instant-Angelo is to provide a useful tool for the community, and we have not yet a plan for publication. However, we intend to create a blog and detailed documentation once I complete the deadline in mid-November.
Currently, only geometry mesh export is supported in Instant-Angelo, but we are working tirelessly to incorporate a new API for exporting textured mesh.
For reference, I have run the example on your provided data. The process takes 2 minutes for structure-from-motion and 20 minutes on 3090 for dense reconstruction. You could download the geometry mesh from this link. Some quick look up results are shown below:
Sorry for the delay on my side, this time. Good luck with your submission!
Thank you so much for running it on our example. This result looks indeed pretty good. Interesting that it creates a hole in the bottom plate while it fills in the space between the feet.
We're currently switching from COLMAP to AliceVision, which is way more modular. Once this is done, we can replace e.g. the SfM/MVS part with a NeRF/NeuS variant. Your method is definitely a hot candidate for that. Thanks!
switching to AliceVision or other methods is too much work for now