bytedance/pv3d

PV3D: A 3D Generative Model for Portrait Video Generation

Zhongcong Xu*, Jianfeng Zhang*, Jun Hao Liew, Wenqing Zhang, Song Bai, Jiashi Feng, Mike Zheng Shou

[Project Page], [OpenReview]

codebase
pretrained checkpoints

Demo Videos

Unconditional Generation

Installation

conda create -n pv3d python=3.7
conda activate pv3d

Install pytorch>=1.11.0 based on your CUDA version (e.g. CUDA 11.3):

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

Install other dependencies:

pip install -r requirements.txt

Install pytorch3d:

pip install "git+https://github.com/facebookresearch/pytorch3d.git"

Download pretrained i3d checkpoints from VideoGPT for Frechet Video Distance evaluation:

gdown 1mQK8KD8G6UWRa5t87SRMm5PVXtlpneJT
mkdir -p ~/.cache/videogpt && mv i3d_pretrained_400.pt ~/.cache/videogpt

Download pretrained ArcFace from InsightFace for multi-view identity consistency evaluation:

wget https://www.dropbox.com/s/kzo52d9neybjxsb/model_ir_se50.pth?dl=0 -O ~/.cache/model_ir_se50.pth

Preparing Dataset

Download the processed voxceleb dataset, uncompress the video clips and camera sequences.

We do not have the plan to release the video clips for other two datasets due to the potential copyright issues. Instead, we provide the list for selected video clips. Please follow the preprocessing pipeline described in our paper to process them.

Training

Start training using the following script and override the arguments:

python3 run.py \
config=configs/projects/pv3d.yaml \
run_type=train \
model=pv3d \
dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.batch_size=16 \
training.num_workers=2 \
model_config.pv3d.density_reg=0.6 \
model_config.pv3d.losses[0].params.loss_vid_weight=0.65 \
model_config.pv3d.losses[0].params.r1_gamma=2.0 \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}

Inference

Download the pretrained checkpoints from release page.

Note: We use a rendering resolution of 64 for all the experiments except for the geometry visualization in qualitative comparison, where we used a rendering resolution of 128. Please download voxceleb_res64 for reproducing the results. As for the teasers, please use the 128 models trained on voxceleb and mixed data.

Generating video clips:

python3 lib/utils/gen_video.py \
config=configs/projects/pv3d.yaml \
run_type=val model=pv3d dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.num_workers=2 \
training.rendering_resolution=${rendering_resolution} \
checkpoint.resume_file=save/${checkpoint} \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}

Evaluation

Compute Frechet Video Distance:

python3 lib/utils/frechet_video_distance/script/scritpt.py --path save/${save_dir}/videos

Generating multi-view results for evaluation:

CUDA_VISIBLE_DEVICES=${gpu} python3 lib/utils/gen_multiview.py \
config=configs/projects/pv3d.yaml \
run_type=val model=pv3d dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.num_workers=2 \
training.rendering_resolution=${rendering_resolution} \
checkpoint.resume_file=save/${checkpoint} \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}

Compute Chamfer Distance:

python3 lib/utils/chamfer_distance/script.py --path save/${save_dir}/multi/depth/

Compute Identity (ID) Consistency:

python3 lib/utils/identity_distance/script.py --path save/${save_dir}/multi/images/

Compute Multi-view Warping Error:

python3 lib/utils/multiview_error/compute_multiview_error.py --path save/${save_dir}/multi --yaw 30 --pitch 0 --size 256

Citation

If you find this codebase useful for your research, please use the following entry.

@inproceedings{xu2022pv3d,
    author = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Junhao and Zhang, Wenqing and Bai, Song and Feng, Jiashi and Shou, Mike Zheng},
    title = {PV3D: A 3D Generative Model for Portrait Video Generation},
    booktitle={The Tenth International Conference on Learning Representations},
    year = {2023}
}

bytedance / pv3d