⭐PyTorch implementation of the paper:
NEF: Neural Edge Fields for 3D Parametric Curve Reconstruction from Multi-view Images [Paper] [Project Page].
Yunfan Ye, Renjiao Yi, Zhirui Gao, Chenyang Zhu, Zhiping Cai, Kai Xu.
- [February 2023] Initial release of code and dataset.
We study the problem of reconstructing 3D feature curves of an object from a set of calibrated multi-view images. To do so, we learn a neural implicit field representing the density distribution of 3D edges which we refer to as Neural Edge Field (NEF). Inspired by NeRF, NEF is optimized with a view-based rendering loss where a 2D edge map is rendered at a given view and is compared to the ground-truth edge map extracted from the image of that view. The rendering-based differentiable optimization of NEF fully exploits 2D edge detection, without needing a supervision of 3D edges, a 3D geometric operator or cross-view edge correspondence. Several technical designs are devised to ensure learning a range-limited and view-independent NEF for robust edge extraction. The final parametric 3D curves are extracted from NEF with an iterative optimization method. On our benchmark with synthetic data, we demonstrate that NEF outperforms existing state-of-the-art methods on all metrics.
This code has been tested with Ubuntu 18.04, one 3080Ti GPU with CUDA 11.4, Python 3.8, Pytorch 1.12, and Pytorch-lightning 1.5.2.
😊Earlier versions may also work~ :)
We introduce ABC-NEF dataset, which contains 115 distinct and challenging CAD models selected from Chunk 0000 of the ABC dataset.
We apply BlenderProc to render multi-view images for each CAD model, and use pre-trained PiDiNet to detect 2D edges.
The entire BlenderProc project used in this work can be downloaded Here. For more details about the ABC dataset, see BlenderProc_NEF/examples/datasets/abc_dataset/main.py for references.
The rendered multi-view images and detected 2D edges are available at Baidu Disk or Google Drive.
The selected original .obj and .feat files are available at Baidu Disk or Google Drive.
The pre-trained models and results are available at Baidu Disk or Google Drive.
To train NEF for a scene, use the following command (example):
(See the pytorch-lighting implementation of NeRF at nerf-pl for reference).
python train.py \
--dataset_name blender \
--root_dir ABC_NEF_examples/00000006 \
--N_importance 64 --img_wh 400 400 --noise_std 0 \
--num_epochs 6 --batch_size 1024 \
--optimizer adam --lr 5e-4 --lr_scheduler steplr \
--decay_step 2 4 8 --decay_gamma 0.5 \
--exp_name 00000006You can also directly run train_command.py to train NEFs, and monitor the training process by tensorboard --logdir logs/ and go to localhost:6006 in your browser.
We provide some pre-trained examples in ckpts_ABC_examples. To extract 3D edge points from trained NEF, use parametric_curve/extract_point_cloud.py. The results should be saved in parametric_curve/ABC_point_clouds.
To extract 3D parametric curves from the 3D edges, use parametric_curve/curve_fitting.py.
We use Chamfer Distance for optimizing the curves. The result curves should be saved in parametric_curve/ABC_curves_result.
To visualize and evaluate extracted parametric curves, run parametric_curve/visualize_and_eval_NEF.py
Install point_cloud_utils to perform evaluations on chamfer distance, precision, recall, F-score and IOU.
Novel view synthesis is not necessary for 3D curves extraction. If there is a need to synthesize novel views of 2D edge maps based on trained NEF models (e.g. for making videos), run the following command:
python eval.py \
--root_dir ABC_NEF_examples/00000006 \
--dataset_name blender --img_wh 400 400 --N_importance 64\
--ckpt_path ckpts_ABC_examples/00000006/epoch=5.ckpt \
--scene_name 00000006or directly run eval_command.py.
@InProceedings{Ye_2023_CVPR,
author = {Ye, Yunfan and Yi, Renjiao and Gao, Zhirui and Zhu, Chenyang and Cai, Zhiping and Xu, Kai},
title = {NEF: Neural Edge Fields for 3D Parametric Curve Reconstruction From Multi-View Images},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {8486-8495}
}