Nature-Cut-Out

This is the python implementation of cutting out a nature contour aroung human body.

Result

I've tested on several images and in most cases the results work pretty well.

Setup

Recommended running environment:

Mac OS X El Capitan (version 10.11.6)
Python 3.6.1

Library:

OpenCV 3.3.0-rc
Scipy 0.19.1
Shapely 1.5.17
math
Numpy 1.13.1
Bezier

Optional library:

descartes
matplotlib

Using pip to install all these library would be recommended:

pip install the-lib-you-want-to-install

Also, if you stuck in some problems when installing OpenCV with Python bindings, I will recommend following this tutorial written by Adrian Rosebrock.

Usage

Basic usage:

python HPE_NatureCutout.py

It will run 27 images in the ./input folder at one time, and show an output image once at a time, press 'esc' to see next output image.

Output:

Output will all be saved to the ./New_Output folder. Each input will generate 3 outputs, including the result simple base on Human-Pose-Estimation, the result after applying Alpha-Shape and the result after using 4-point Bézier curve.

Run on your own images:

Please first go to this website and scroll down to the bottom to download the matlab code on constructing 2D and 3D human pose. Save the 2D human pose result as the .mat file:

filename = 'testImg21'
fname = strcat('./pred_2d',filename(8:end));
save(fname,'preds_2d');

which 'preds_2d' is the parameter's name.

Put both your .mat files and images in the input folder, and make sure the format and the name of files are the same as mine in the input folder.

Algorithm

The whole process could be divided into the following process:

1. First using 2D human pose estimation to get the landmarks of a human body.

Big thanks to the great work done by X. Zhou, M. Zhu, S. Leonardos, K. Daniilidis., Download the code from the website.

generate heatmap in advance:

th ./pose-hg-demo/run-hg.lua

run the matlab code to get the 16 landmarks(shown in right most image):

2. Base on 2D landmarks, calculate those possible contour points.

Base on the vector constructed between 2 landmarks, calculate its norm direction and mark out these points(green points in the concept figure) as the possible contour points. The following are the concept figure:

The result on the above example image would be just like:

3. Apply alpha shape to find out those key points which will contribute to build the contour.

Thanks to the clear tutorial by Sean Gillies and KEVIN DWYER, you can click the links to look into detail. My alpha shape function is mainly built on the code shown in the above two links.

The whole concept is first build delaunay triangulation base on SciPy library, and apply alpha-shape to remain those vetexes whose triangle's radius of the circumcircle is small enough. Finally we can extract all these points as the control points to build a smooth contour later on.

The following pictures are the result from step 2, Delaunay-triangle, Alpha-shape and the exterior points of the alpha shape respectively.

4. Interpolating between key points and apply 4-point Bézier curve to reconstruct the nature-cut-out.

First interpolate 30 points between two neighbor points, and use the bezier-function in python package to reconstruct a more smooth contour.

The following pictures are the result from step 3, the result after interpolation and the result after applying bezier curve shown in 100 points and 500 points, respectively.

cardinalblue / Nature-Cut-Out