TangZJ / DMRF

Dynamic Mesh-Aware Radiance Fields (ICCV2023): Raytracing rendering and interactive simulating mesh with NeRF

Dynamic Mesh-Aware Radiance Fields

Yi-Ling Qiao*, Alexander Gao*, Yiran Xu, Yue Feng, Jia-Bin Huang, Ming C. Lin

[Project] [Paper]

Introduction

This is an implementation of the ICCV 2023 paper Dynamic Mesh-Aware Radiance Fields. The NeRF volume rendering is largely based on NVIDA's great work InstantNGP. The physics part also runs with NVIDIA's Warp and/or DeepMind's Mujoco. A more detailed tutorial for ray tracing can be found in Ray Tracing in One Weekend.

Structure

We have now uploaded all core C++/CUDA files but are still cleaning the python scripts, config files, and datasets for running each experiment.

The entry to the main rendering loop for raytracing-NeRF coupling is the render_nerf_rt function at ./src/testbed_nerf.cu.

The folder ./simpleRt/ also has CUDA-based raytracing-related functions.

Physics simulation module will be called in python scripts.

Setup

This project has been tested with Ubuntu 20.04, gcc 9.4.0, CUDA 11.8.

git clone [e-mail]:YilingQiao/DMRF.git
cd DMRF
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make -j
cd ..

If you have any issues building/running this project, please try to make sure you can run InstantNGP first (especially its commit c4d622e, which is a start point of our project).

TODOs

1. An end-to-end tutorial for training NeRF, estimating light source and geometry, inserting mesh, and rendering.

2. An introduction to the config file system and code structure.

3. Other scenes presented in the paper.

Demos

Let's download the data and pretrained NeRF from this google drive.

Put them under DMRF/extra_data.

Reflective ball

We can insert a reflective ball into the nerf360/garden scene. The ball will also cast shdows onto the ground.

Scene desctiption of the ball, lightsource, and shadow mesh can be found in the configuration file ./scripts/exp/garden_ball.json".

python ./scripts/exp_garden_ball.py --mode nerf --load_snapshot ./extra_data/nerf360/garden/35000.msgpack  --width 800 --height 600 --gui

Infinite mirror room

Let's go one step further and add more objects into the scene.

With relfective walls around, now the ray can bounce multiple times inside a mirror room.

Since the scene is complex, we choose not to render it in GUI. Images will be saved to ./render_output/garden_mirror/.

python ./scripts/exp_garden_mirror.py --mode nerf --load_snapshot ./extra_data/nerf360/garden/35000.msgpack --width 200 --height 150 --video_spp 4 

Interactive game

We can also add a controllable object into NeRF and see how it interact with the scene.

Here we use Warp to compute the dynamics. You might need to install this package first.

Use W/S/A/D/Z/X to control the ball's moving direction. Use space to stop the ball

python ./scripts/exp_game.py --mode nerf --load_snapshot ./extra_data/nerf360/kitchen/35000.msgpack --width 800 --height 600 --gui --simulation 

Run with your own data

Preprocess

You can refer to this tutorial for preprocessing your own data.

Config file

TODO. An explanation of the config file system.

BibTex

@article{qiao2023dmrf,
  author    = {Yi-Ling Qiao and Alexander Gao and Yiran Xu and Yue Feng and Jia-Bin Huang and Ming C. Lin},
  title     = {Dynamic Mesh-Aware Radiance Fields},
  journal   = {ICCV},
  year      = {2023},
}