Human-Centric Super-Resolution based on 3D texture model

Description

The reposity contains the code that generates the human models from high-definition RGB images or videos, binds the detailed texture onto the model, and renders it to low-definition frames according to refined human 3D pose.

Firstly, the project use SMPL as basic 3D parameterized model, the OpenPose, PSPNet and the HMR as the supervision of minimizing the error of re-projection of SMPL. Combined with temporal contraint, it can produce a reasonable initial result.

For the mesh reconstruction, the contour of SMPL is expaneded to fit the human silhouette in image to generate new SMPL template. The Multi-label $\alpha$-expansion is used to solve a discrete optimition process to find the correspondence between the contours of SMPL and human silhouette. To reduce the peak noise in edge, the laplacian coordination is used to preserve the smoothness of edge.

For the motion refinement, the high frequency details from high-definition motion and low frequency long-term trend from low-definition motion are fused to generate the final results.

The project use opendr as a simple rendering tool. But it may produce a low-quality texture. Alternatively, the OpenGL is used to generate the final texture. It will upload soon.

Data Preprocessing

The high and low definition images can be captured with Double Camera System. The best way is to crop the human using detection boxes in each frame. YOLO and KCF are recommended to generate the feasible data.

Installation

conda create -n conda_name python=2.7
pip install -r requirements.txt
pip install pip==8.1.1
pip install opendr==0.78
pip install --upgrade pip

Demo

Because of the low ability to show off high-resolution pictures in GitHub, it is difficult to check the super-resolution results. For convenience, only the results of model reconstrution are shown here.

There are some rendering result in MPII dataset. The original low-definition human sequence is $\times$ 8 downsampled.