🧙♀️ The name is inspired by repairing charm in Harry Potter.
REPARO is a novel approach for compositional 3D asset generation from single images. It employs a two-step process: first, it extracts individual objects from the scene and reconstructs their 3D meshes using off-the-shelf imageto-3D models; then, it optimizes the layout of these meshes through differentiable rendering techniques, ensuring coherent scene composition. By integrating optimal transport-based long-range appearance loss term and high-level semantic loss term in the differentiable rendering, REPARO can effectively recover the layout of 3D assets. The proposed method can significantly enhance object independence, detail accuracy, and overall scene coherence.
- 🔥 Release on arxiv!
- 🔥 Release demo page!
- 🔥 Release test dataset GSO 20 selected from Google Scanned Objects!
- Release synthetic single assets generated by DreamGaussian and TripoSR.
- Release differentiable layout alignment code based on 6D-RGBDXY inverse rendering.
- Release metrics evaluation scripts in the light of multi-view of assets.
- Release more qualitative experimental results.
☀️ We are truly indebted to the following outstanding open source work upon which we have based our work: SAM, DreamGaussian, Wonder3D, OpenLRM, DROT, TripoSR, stablediffusion-infinity, and nvdiffrast.
@misc{han2024reparo,
title={REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignment},
author={Haonan Han and Rui Yang and Huan Liao and Jiankai Xing and Zunnan Xu and Xiaoming Yu and Junwei Zha and Xiu Li and Wanhua Li},
year={2024},
eprint={2405.18525},
archivePrefix={arXiv},
primaryClass={cs.CV}
}