sxyu/avatar

AR/VR Avatar Project: Fitting SMPL body model to depth data in real time on CPU (Fall 2019)

Demo video: https://drive.google.com/file/d/1KQ0g_R77x80c6WbFKTXefvsNO9F1ITxW/view?usp=sharing

Contains

SMPL model loader and representation in C++ (Avatar)
Fast SMPL parameter optimizer (wrt. a point cloud) based on Ceres-solver (AvatarOptimizer)
Real-time human body segmentation system using random forest, with weights provided (RTree)
- Custom random forest implementation and parallelized training system provided
Basic first-frame background subtraction system (BGSubtractor)
Gaussian mixture model to inform likely poses, as in SMPLify (GaussianMixture)
DepthCamera interface from OpenARK (AzureKinectCamera/DepthCamera)
C++ Vicon skeleton (asf/amc) loader to pose and animate the SMPL model
Other miscellaneous utilities such as data recording

A smaller reimplementation of OpenARK Avatar using only analytic derivatives.

Building

Dependencies

Boost 1.58
OpenCV 3.3+ (OpenCV 4 not supported)
Eigen 3.3.4
Ceres Solver 1.14 (Ceres 2 not supported).
- This is very performance critical, and it is strongly recommended to manually build Ceres with LAPACK and OpenMP support.
- If you are using an Intel processor, it is also recommended to use MKL as BLAS/LAPACK. Otherwise ATLAS is recommended.
- Finally, make sure you build Ceres in release mode.
zlib, for reading SMPL npz model
One of (optional but required for live-demo)
- K4A (Azure Kinect SDK)
- libfreenect2
PCL 1.8+, optional

Earlier versions of these libraries may work, but I have not tested them

How to build

If you haven't already, install CMake.

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j4

Replace 4 with an appropriate number of threads. Add -DWITH_PCL=ON to enable PCL, add -DWITH_K4A=OFF to disable looking for Azure Kinect SDK, add -DBUILD_RTREE_TOOLS=OFF to disable building RTree tools such as rtree-train, rtree-run-dataset.

For unknown reasons, sometimes I encounter linker errors when not manually linking OpenMP. If this happens configure with -DWITH_OMP=ON.

Outputs

Core

live-demo: from live-demo.cpp. Live demo, runs the system end-to-end on Azure Kinect camera input. Requires K4A library to be installed
demo : from demo.cpp. Runs the system end-to-end on an OpenARK dataset in standard format (depth_exr, etc)
data-recording : from DataRecording.cpp. Tool for recording datasets from the Azure Kinect camera. Mostly copied from OpenARK, but fixes memory bug.
libsmplsynth.a : the static library which the above depend on. I configure the project like this to improve build times when editing different outputs.

SMPL Model Tools

smplsynth : from smplsynth.cpp. Synthetic human dataset generator
smpltrim : fom smpltrim.cpp. A tool for generating partial SMPL models, including creating a smaller model with a specific joint as root, or cutting off limbs

Random Forest Tools

rtree-train: from rtree-train.cpp. High performance random tree trainer. Find trained trees in releases on Github
rtree-transfer: from rtree-transfer.cpp. Tool to refine a trained random tree by recomputing leaf distributions over a huge amount of images.
rtree-run: from rtree-run.cpp. Run rtree on images (not important).
rtree-run-dataset: from rtree-run-dataset.cpp. Run rtree on OpenARK dataset in standard format (depth_exr, etc)

Miscellaneous

scratch : from scratch.cpp. Currently configured to show human avatar when ran, with (limited) options to adjust pose and shape. Generally, used for scratch.
optim : from optim.cpp. Currently disabled since not updated after API change; optimizes avatar pose to fit a synthetic point cloud.

Getting model data

New please see Github releases on how to get model data. https://github.com/sxyu/avatar/releases

License

Apache 2.0

sxyu / avatar