SurfelWarp

SurfelWarp is a dynamic reconstruction system similar to DynamicFusion. Compared to other dynamic reconstruction methods, surfelwarp uses flat surfel array (instead of volumetric field) as the geometry representation, which makes the pipeline robust and efficient. The approach is described in our paper.

Demo [Video][Presentation]

Publication

Wei Gao and Russ Tedrake, "SurfelWarp: Efficient Non-Volumetic Single View Dynamic Reconstruction", Robotics: Science and Systems (RSS) 2018 [Project][Paper][Presentation]

Build Instruction

The code is developed on Visual Studio 2015 with CUDA 9 . Note that CUDA 10 and higher versions are not compatible with pcl 1.8 and don't work. The code also works with Ubuntu 16.04, but for unknown reason it runs much slower on ubuntu (seems to be problem with GPU driver that only permits Debug mode). You might need to update these lines of the build script according to your GPU (the default is for Nvidia Titan Xp).

In addition to cuda, the code depends on pcl , opencv and glfw. On Ubuntu, these dependencies can be installed with apt, while they need to be built from source on windows. For Ubuntu 16.04, you can run the following command to install the dependency

sudo apt-get install libpcl-dev libopencv-dev libglfw3 libglfw3-dev

Now you are ready to build

git clone https://github.com/weigao95/surfelwarp
cd surfelwarp
mkdir build && cd build
cmake ..
make

For build on windows please refer to this document. We also provide a pre-built binary for the windows platform (The arch flag for this executable is sm_60).

Run Instruction

We use the VolumeDeform dataset to illustrate how to run the code. An example configuration file is provided here for the "boxing" sequence. First, you need to download the boxing sequence from the VolumeDeform dataset and extract it to data_root, your file structure should look like

${data_root}/frame-000000.color.png
${data_root}/frame-000000.depth.png
...

You also need to download the trained model for Global Patch Collider (gpc) from here. Let the path to the model be ${gpc_path} .

In the configuration file, please modify the data_prefix and gpc_model_path to ${data_root} and ${gpc_path}, respectively. After that, you can run the algorithm with

cd ${project_root}/build/apps/surfelwarp_app
./surfelwarp_app /path/to/config

If everything goes well, the executable would produce the reconstructed result per frame in the same folder as surfelwarp_app.

FAQ

What's the typical speed of surfelwarp?

On the windows platform with a Nvidia Titan Xp GPU, the processing time is usually less than 20 [ms] per frame. To test the speed in surfelwarp_app, you need to build with Release and disable offline rendering (which takes most of the time).
How to use my own data?

You might override the FetchInterface and use it with ImageProcessor. If you don't have performance requirement, you can also convert your data into the format of VolumeDeform.
How to deal with topology change?

Currently, only periodic reinitialization is implemented. To use it, set this flag in config file. More advanced criterion of reinitialization would come soon.

TODO

The code is re-factored and improved from the repo of our RSS paper. There are some planned new features and some old code need to be ported into this repository. Here is a list of TODOs:

Add sparse feature correspondence
Implement albedo reconstruction
Rewrite automatic reinitialization detection for the new geometry and loss function. Currently, only periodic reinitialization is implemented
Port the GPU kdtree querying (although the overall speed is already relatively fast)

Contact

If you have any question or suggestion regarding this work, please send an email to weigao@mit.edu

jbyu / surfelwarp