nsalminen / HMDRemoval

Source code of the framework proposed in the paper "Generative RGB-D Face Completion for Head-Mounted Display Removal", presented at the VHCIE workshop at IEEE VR 2021.

Generative RGB-D Face Completion for Head-Mounted Display Removal

An open source framework for generative HMD removal in RGB-D images as described in the paper "Generative RGB-D Face Completion for Head-Mounted Display Removal". This framework builds on the RGB image inpainting framework proposed in "Free-Form Image Inpainting with Gated Convolution" by Yu et al.

The source code of this framework is based on the source code by Yu et al.

Qualitative results summary. Shown for color (RGB), depth (D), and estimated surface normals (SN). For visualization, D is normalized to [0, 1] and displayed with the inferno colormap from the matplotlib package. The normal vectors (x, y, z) for each pixel in SN are estimated based on D and are visualized with RGB values.

Run

0. Requirements:
   • Install Python (v3.6).
   • Install the requirements listed in requirements.txt.
     • Run pip install -r requirements.txt to install with pip.
1. Training:
   • Prepare training images file list with create_flist.py (needs modification based on file name structure).
   • Modify inpaint.yml to set DATA_FLIST, LOG_DIR, IMG_SHAPES and other parameters.
   • Run python train.py.
2. Resume training:
   • Modify MODEL_RESTORE flag in inpaint.yml. E.g., MODEL_RESTORE: hmdRemoval_341953.
   • Run python train.py.
3. Testing:
   • Run python test.py --image data/input.png --mask data/mask.png --reference data/reference.png --output examples/output.png --checkpoint logs/hmdRemoval_341953.
4. Batch testing:
   • Download facenet_keras.h5 and place in repo directory.
   • Run python batch_test.py --summary 1 --flist data/rgbd_faces/data_flist/test_face_mask_ref.flist --image_height 224 --image_width 224 --output_dir output --checkpoint_dir /logs/hmdRemoval_341953
5. Evaluation:
   • Run batch test (see point 4).
   • Run python evaluate.py --pred_folder output --flist data/rgbd_faces/data_flist/test_face_mask.flist
6. Still have questions?
   • If you still have questions (e.g.: How to use multi-gpus? How to do batch testing?), please first search over closed issues in the repo of the base framework. If the problem is not solved, please feel free to open an issue.

TensorBoard

Visualization on TensorBoard for training and validation is supported. Run tensorboard --logdir model_logs --port 6006 to view training progress.

License

CC 4.0 Attribution-NonCommercial International

The software is for educational and academic research purpose only.

Citing

@inproceedings{numan2021generative,
  title={Generative RGB-D Face Completion for Head-Mounted Display Removal},
  author={Numan, Nels and ter Haar, Frank and Cesar, Pablo},
  booktitle={2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)},
  pages={109--116},
  year={2021},
  organization={IEEE}
}