This repository provides tools for preprocessing videos for HDTF dataset used in the paper
We are still refining the environment setup at the moment. Please refer to the README file in each subdirectory for guidance. Once we finalize everything, we will gather all the necessary requirements into a requirements.txt file located in the root directory. You can create a conda enviroment first via
conda create --name otavatar_processing python=3.9
This section is adopted from HDTF
This seciton is adopated from MODNet
This section is in the FaceCrop folder, adopted from FOMM. The primary objective is to zoom in on the facial regions and discard any irrelevant background.
This section is adopted from PIRenderer
This section is adopted from PIRenderer
This section is adopted from ADNeRF
During our attempt to remove the background from uncropped videos, we observed that the matting results were unstable between frames. This was due to the fact that the model was trained on images where the head only occupied a small portion of the entire image, which is precisely what is captured in raw video. As a result, we decided to first remove the background and then crop the face, which led to more encouraging results.
Initially, we planned to implement [FOMM]((https://github.com/AliaksandrSiarohin/video-preprocessing), which has been widely utilized in numerous prior methods, as a means of talking face processing. However, we encountered difficulties in obtaining a stable and accurate head pose during the fine stage, as the head occupied too much space in the image. As a result, we opted to make some minor adjustments to our approach.
We appreciate HDTF, FOMM, ADNeRF, PIRenderer, MODNet for providing their processing script.