Rendering, Animating and Meshing Actors with NeRF
A library for rendering neural actors, and benchmarking dynamic NeRF
Examplar
Point Cloud Completion
Neural Rendering
Dynamic Meshing
SMPL Fitting
Setup
Notice that working on NVidia 3090 requires certain pytorch and torchvision versions
pip install torch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
Or download the wheel and install with in case pytorch site is not included in the pip source
pip install torch-1.11.0+cu113-cp37-cp37m-linux_x86_64.whl
As for COLMAP, version 3.8 (the lastest in early 2023) has problen with CUDA 11.X, version 3.7 fits the best for SIFT feature extraction and matching with CUDA.https://github.com/colmap/colmap/tree/3.7
Blender
/snap/bin/blender ~/Downloads/rp_aliyah_4d_004_dancing_BLD/rp_aliyah_4d_004_dancing_2k.blend --background --python 'render/renderpeople.py' -- --with_images --start 1 --end 150
Format the camera parameters
python3 format_convert/json2yaml.py ~/Documents/datasets
Then add %YAML:1.0
as header for the newly generated yml file so that OpenCV can parse it
:%s/-\ \ //g
:%s/\ \ \ \ \ \ /\ \ \ /g
Mocap
Detect hand and face keypoints using OpenPose
python3 scripts/preprocess/extract_video.py ~/Documents/datasets --openpose ~/Downloads/openpose --handface --ext png --with_img --end 1
python3 apps/demo/mv1p.py ~/Documents/datasets --out ~/Documents/datasets/output/smpl --vis_det --vis_repro --undis --vis_smpl --end 1
Citation
@misc{rama2023wang,
Author = {Yida Wang},
Year = {2023},
Note = {https://github.com/wangyida/neural-actor},
Title = {Rendering, Animating and Meshing Actors with NeRF}
}