OloOcki / scan2lod3

Implementation of the CVPR paper "Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks"

🚨 Scan2LoD3 🏙️

Implementation of the CVPR Workshops '23 paper:

"Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks"

Scan2LoD3: Our method reconstructs detailed semantic 3D building models; Its backbone is laser rays’ physics providing geometrical cues enhancing semantic segmentation accuracy.

🌟 Highlights 🌟

The workflow of the proposed Scan2LoD3 consists of three parallel branches:

• The first is generating the point cloud probability map based on a modified Point Transformer network (top);
• the second is producing a conflicts probability map from the visibility of the laser scanner in conjunction with a 3D building model (middle);
• and the third is using Mask-RCNN to obtain a texture probability map from 2D images. We then fuse three probability maps with a Bayesian network to obtain the final facade-level segmentation, enabling a CityGML-compliant LoD3 building model reconstruction.

👎 BEFORE scan2lod3: LoD2

👍 AFTER scan2lod3: LoD3

🔎 Our approach to visibility analysis:

Visibility analysis using laser scanning observations and 3D models on a voxel grid. The ray is traced from the sensor position si to the hit point pi. The voxel is: empty if the ray traverses it; occupied when it contains pi; unknown if unmeasured; confirmed when occupied voxel intersects with vector plane; and conflicted when the plane intersects with an empty voxel.

👷 Implementation overview

The implementation can be divided into several steps:

1. Ray casting (C++)
2. Mask-RCNN (Python)
3. Point Transformer (Python)
4. Confidence estimation; probability map projection;
5. Bayesian network estimate (R)
6. Shape extraction (FME)
7. CityGML-compliant 3D modeling (FME)

🎓 Paper

For the in-depth conept understanding do not hesitate to check out the paper:

@inproceedings{wysocki2023scan2lod3,
  title={Scan2LoD3: Reconstructing semantic 3D building models at LoD3 using ray casting and Bayesian networks},
  author={Wysocki, Olaf and Xia, Yan and Wysocki, Magdalena and Grilli, Eleonora and Hoegner, Ludwig and Cremers, Daniel and Stilla, Uwe},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={6548--6558},
  year={2023}
}

💽 Data

Small sample dataset it attached to this repo: \raycasting\examples. For the vector objects and point cloud please check the tum2twin (soon available!) benchmark dataset.

📫 Contact details

Should you have any further questions do not hesitate to drop me an email: [e-mail]