HannesStark / SMPL-NeRF

Embed human pose information into neural radiance fields (NeRF) to render images of humans in desired poses :running: from novel views

PyTorch Neural Radiance Fields (NeRF) with SMPL embedding

Video | Paper

This repository is a PyTorch implementation of NeRF which can be trained on images of a scene to then render novel views. To enjoy the the vanilla NeRF just run the train file with model_type=nerf. Additionally, different model types are supported that embed an SMPL model of a human to control its pose in addition to the view point.

Quickstart

• Create a 128x128 synthetic dataset of humans with different arm angles for the model type smpl_nerf.

python create_dataset.py --dataset=smpl_nerf --save_dir=data --resolution=128 --start_angle=0 --end_angle=1 --number_steps=1 --human_number_steps=10 --multi_human_pose=1 --human_start_angle=0 --human_end_angle=60

• Install torchsearchsorted.

cd torchsearchsorted
pip install .

• Run the train file.

python train.py --experiment_name=SMPLNeRF --model_type=smpl_nerf --dataset_dir=data --batchsize=64 --batchsize_val=64 --num_epochs=100 --netdepth=8 --run_fine=1 --netdepth_fine=8

• Start tensorboard.

tensorboard --logdir=logs/summaries --port=6006

Navigate to localhost:6006 in your browser and watch the model train.

Requirements

• PyTorch >=1.4
• matplotlib
• numpy
• imageio
• configargparse
• torchsearchsorted
• smplx

Creating synthetic datasets requires

• pyrender
• trimesh

Setting up the Baseline

• Clone Pix2Pix repo (TODO: install dependencies):

mkdir baseline/
cd baseline/
git clone https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.git
cd ..

• Create Dataset with (RGB, depth)-pairs

python create_dataset.py --dataset=pix2pix --save_dir=baseline/pytorch-CycleGAN-and-pix2pix/datasets/smpl --resolution=128 --start_angle=-90 --end_angle=90 --number_steps=10

• Train Pix2Pix on datasets (set name for experiment, set gpu_ids=-1 for CPU)

cd baseline/pytorch-CycleGAN-and-pix2pix/
python train.py --gpu_ids=0 --model=pix2pix --dataroot=datasets/smpl --name=SMPL_pix2pix --direction=BtoA --save_epoch_freq=50

Model Types

• nerf: the vanilla NeRF
• image_wise_dynamic: the rays are not shuffeled between the image during processing so that the warp for every ray is only calculated once.

NeRF is from the Paper:

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, & Ren Ng. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.