Giters

lcybuzz / Personal-Face-Related-Paper-Record

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Face Related Paper Record

Under construction！

Table of Contents

Rank

  • Face Parsing
    • ★★★
    • ★★
      [MO-GC], [Guided by Detected]
    • ★
      [CnnRnnGan], [RNN-G], [FC-CNN], [Adaptive Receptive Fields], [Exemplar-Based], [RED]
  • Face Detection
    • ★★★
    • ★★
      [MTCNN], [SSH]
    • ★
  • Landmark Detection
    • ★★★
    • ★★
      [DAN], [LAB]
    • ★

Face Parsing

Deep Learning Methods

Hierarchical face parsing via deep learning

[Paper] Hierarchical face parsing via deep learning
[Year] CVPR 2012
[Author] Ping Luo, Xiaogang Wang, Xiaoou Tang
[Pages]
[Description]

MO-GC ★★

[Paper] Multi-Objective Convolutional Learning for Face Labeling
[Year] CVPR 2015
[Author] Sifei Liu, Jimei Yang, Chang Huang, Ming-Hsuan Yang
[Pages] https://www.sifeiliu.net/face-parsing
[Description]

  1. 模拟CRF, 提出一种用多个目标函数优化一个CNN的人脸解析方法. 一个loss针对unary label likehood, 一个loss针对pairwise label dependency;
  2. 提出一种nonparametric prior作为global regularization. 首先在脸部key point真值图像块上基于PCA建立一形状子空间, 测试时搜索与测试图像最相似的若干真值图像, 根据key point将真值图像与测试图像align，将几张aligned后的mask取平均作为prior;
  3. 在LFW和Helen上实验, 多目标函数的策略对精度有微小提升, nonparametric prior效果提升明显：

Guided by Detected ★★

[Paper] A CNN Cascade for Landmark Guided Semantic Part Segmentation
[Year] ECCV 2016 workshop
[Author] Aaron S. Jackson, Michel Valstar, Georgios Tzimiropoulos
[Pages] http://aaronsplace.co.uk/papers/jackson2016guided/index.html
[Description]

  1. 提出一种用landmarks引导part segmentation的方法, 用pose-specific信息辅助分割, 分为landmark检测和分割两步;
  2. landmark detection: 先用一个FCN预测68个高斯状的landmarks(68个输出channel,每个channel对应1个2D Gaussian)
  3. segmentation: 将detection得到的68个channel加到输入图像上, 再用1个FCN完成分割. 这个的一个key aspect是验证集上的landmark localization error加到landmark真值上去生成2D Gaussian (没看懂他的理由???)
  4. 实验部分用IoU评价, 但是没与其它方法对比, 说服力略显不足; 数据是自行从landmark数据集中生成的分割图.

CnnRnnGan ★

[Paper] End-to-end semantic face segmentation with conditional random fields as convolutional, recurrent and adversarial networks
[Year] arXiv 1703
[Author] Umut Güçlü, Yagmur Güçlütürk, Meysam Madadi, Sergio Escalera, Xavier Baró, Jordi González, Rob van Lier, Marcel van Gerven
[Pages] https://github.com/umuguc (还没开源)
[Description]

  1. 大致浏览. 本文提出了一个大杂烩, 将dilation, CRFasRNN, adversarial training整合到一end to end的框架中. 不过, 首先要检测landmark, 将landmark连接生成初始分割图, 再用landmark将输入图像和分割图与模板对齐.
  2. 效果较好, 但暂时未开源.
  3. 有一个问题没细看: 在Helen上实验时, 是分别训练了5个网络解析不同类别吗??

RNN-G ★☆

[Paper] Parsing via Recurrent Propagation
[Year] BMVC 2017
[Author] Sifei Liu, Jianping Shi, Ji Liang, Ming-Hsuan Yang
[Pages]
[Description]

  1. 提出一种coarse to fine的人脸解析方法, 第一阶段解析出皮肤, 头发和背景, 第二部解析facial components. CNN和RNN参数都较少, 因此速度很快.
  2. 第一阶段将CNN的hierarchical representation和RNN的label propagation结合起来. CNN有两个输出, 一个是feature map, 另一个是edge map. RNN考虑上下左右四个方向, 以feature map为输入, 并用edge map作为gate, 即边缘处两个node的联系应该小, 相同类别区域两个node联系应该大.
  3. 第二个阶段设计了eye/eyebrow, nose和mouth三个子网络, 根据landmark将五官crop成patch, 送入相应的网络进行解析.
  4. 本文也需要额外的landmark检测, 检测出的landmard用于将脸转正和crop五官.

FC-CNN ★☆

[Paper] Face Parsing via a Fully-Convolutional Continuous CRF Neural Network
[Year] arXiv 1708
[Author] Lei Zhou, Zhi Liu, Xiangjian He
[Pages]
[Description]

  1. 将CRF与CNN结合起来, CRF的思路应该是来源于MO-GC, 模型包括unary, pairwise和continuous CRF(C-CRF)三个子网络; 网络基于Caffe, 可以端到端训练. 未开源,性能较好.
  2. Unary net采用类似SegNet的结构. pairwise net将相邻像素的feature连接起来并用12和21的卷积得到其水平和垂直方向的相似的, 最后得到相似度矩阵.
  3. C-CRF网络首先用superpixel pooling layer将unary和pairwise网络的pixel-level feaature转化为region-level feature. 目的是保留边界信息和保证同区域标注的一致性(?). 再使用unary和pairwise的超像素特征构成目标能量函数.
  4. 介绍了一种端到端训练C-CRF的方法, 没细看.
  5. 貌似应该需要额外的方法得到超像素.

Adaptive Receptive Fields ★

[Paper] Learning Adaptive Receptive Fields for Deep Image Parsing Network
[Year] CVPR 2017
[Author]   Zhen Wei, Yao Sun, Jinqiao Wang, Hanjiang Lai, Si Liu
[Pages]
[Description]

  1. 提出学习一个参数f, 对feature map进行缩放, 从而自适应地改变感受野大小.
  2. 设计一个multi-path模型, 为打破各支路的均衡性, 使用了loss guidance, 即对某一支加大某些类的权重, 如把类别分为{eye, eyebrow}和{nose, lip, mouth}两组, 用起分别对不同支路加权. 这样能引导各个分支学习到适合分割特定目标的感受野.
  3. loss guidance的思路可以借鉴, 但从结果来看多个支路的精度反而不如单支路的...
  4. 在一个数据集学到的参数f, 应该是只适应于当前任务, 感觉不太适用于模型迁移?

RED ★

[Paper] Residual Encoder Decoder Network and Adaptive Prior for Face Parsing
[Year] AAAI 2018
[Author]   Tianchu Guo, Youngsung Kim, Hui Zhang, Deheng Qian, ByungIn Yoo, Jingtao Xu, Dongqing Zou, Jae-Joon Han, Changkyu Choi
[Pages]
[Description]

  1. 基于DeconvNet网络结构的人脸解析方法, online计算先验.

Classical Methods

Exemplar-Based ★

[Paper] Exemplar-Based Face Parsing
[Year] CVPR 2013
[Author] Brandon M. Smith, Li Zhang, Jonathan Brandt, Zhe Lin, Jianchao Yang
[Pages] http://pages.cs.wisc.edu/~lizhang/projects/face-parsing/
[Description]

  1. 粗读, 基于exemplar的人脸解析. 提供了一个基于Helen的人脸解析数据集

Face Detection

Cascade CNN

[Paper] A Convolutional Neural Network Cascade for Face Detection
[Year] CVPR 2015
[Author] Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, Gang Hua
[Pages] https://github.com/anson0910/CNN_face_detection (Unofficial)
[Description]

Faceness-Net

[Paper] From Facial Parts Responses to Face Detection: A Deep Learning Approach
[Year] ICCV 2015
[Author] Shuo Yang, Ping Luo, Chen Change Loy, Xiaoou Tang
[Pages] http://shuoyang1213.me/projects/Faceness/Faceness.html
[Description]

MTCNN ★★

[Paper] Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
[Year] SPL 2016
[Author] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, Yu Qiao
[Pages] https://kpzhang93.github.io/MTCNN_face_detection_alignment/
[Description]

  1. 以3个CNN级联的方式，完成coarse到fine的人脸检测和对齐;
  2. 三个网络分别为Prposal(P)-Net, Refine(R)-Net和Output(O)-Net, 三个网络都是结构相似的小型CNN，总体速度较快;
  3. 网络的训练包括三个task: 人脸分类(是否是人脸的二分类问题), bounding box回归, landmark定位. 三个任务是分别取样本和训练的;
  4. 提出online hard sampling mining, 在一个mini-batch中对每个sample的loss排序, 只取loss由大到小前70%的sample参与back propagation

SSH ★★

[Paper] SSH: Single Stage Headless Face Detector
[Year] ICCV 2017
[Author] Mahyar Najibi, Pouya Samangouei, Rama Chellappa, Larry S. Davis
[Pages] https://github.com/mahyarnajibi/SSH
[Description]

  1. Single stage, no head of classification network
  2. Scale-invariant by design, detect faces from various depths

Finding Tiny Faces

[Paper] Finding Tiny Faces
[Year] CVPR 2017
[Author] Peiyun Hu, Deva Ramanan
[Pages] http://www.cs.cmu.edu/~peiyunh/tiny/index.html
[Description]

Landmark Detection

CNN_FacePoint

[Paper] Deep Convolutional Network Cascade for Facial Point Detection
[Year] CVPR 2013
[Author] Yi Sun, Xiaogang Wang, Xiaoou Tang
[Pages] http://mmlab.ie.cuhk.edu.hk/archive/CNN_FacePoint.htm
[Description]

TCDCN

[Paper] Facial Landmark Detection by Deep Multi-task Learning
[Year] ECCV 2014
[Author] Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang
[Pages] http://mmlab.ie.cuhk.edu.hk/projects/TCDCN.html
[Description]

3DDFA

[Paper] Face Alignment Across Large Poses: A 3D Solution
[Year] CVPR 2016
[Author] Hai Zhu, Lei Zhen, Xiaoming Liu, Hailin Shi, Stan Z Li
[Pages] http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm
[Description]

DAN ★★

[Paper] Deep Alignment Network: A convolutional neural network for robust face alignment
[Year] CVPRW 2017
[Author] Marek Kowalski, Jacek Naruniec, Tomasz Trzcinski
[Pages]
https://github.com/MarekKowalski/DeepAlignmentNetwork (Official, Theano)
https://github.com/zjjMaiMai/Deep-Alignment-Network-A-convolutional-neural-network-for-robust-face-alignment (TF)
https://github.com/mariolew/Deep-Alignment-Network-tensorflow (TF)
[Description]

  1. 级联的人脸对齐方法, 思路简洁, 速度快.
  2. 每个stage的输入由三部分组成: 输入图像, 上一个stage的landmark heatmap和feature, 值得注意的是, 算法使用上阶段的landmark和标准脸计算变换矩阵, 并对三个输入做了对齐, 这使得算法对大姿态人脸有较好的鲁棒性.

LAB ★★

[Paper] Look at Boundary: A Boundary-Aware Face Alignment Algorithm
[Year] CVPR 2018
[Author] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, Qiang Zhou
[Pages] https://wywu.github.io/projects/LAB/LAB.html
[Description]

  1. 利用边界信息和对抗训练做人脸关键点检测, 效果很好, 对大姿态表情鲁棒性较强.
  2. 模型由Boundary-Aware关键点回归, Boundary Heatmap预测和Landmark-Based Boundary Effectiveness Discriminator三部分组成. 关键点回归采用多级boundary heatmap融合策略; 边界heatmap预测的是像素到边界的距离, 并使用了message passing机制; 对抗训练部分使用一判别器判断boundary heatmap的有效性, 并为整个网络引入一对抗loss
  3. 开源了基于caffe的测试代码. 感觉方法的训练包括很多细节, 复现性能可能有些难度.

Others

CBN ★

[Paper] Deep Cascaded Bi-Network for Face Hallucination
[Year] ECCV 2016
[Author] Shizhan Zhu, Sifei Liu, Chen Change Loy, Xiaoou Tang
[Pages] https://github.com/zhusz/ECCV16-CBN
[Description]

  1. 大致浏览, 人脸增强类似于超分辨率重建, 但利用了人脸的结构信息. 设计了一gated deep bi-network, 一个分支是common branch即普通的残差预测网络, 一个分支是high-frequency branch利用dense field得到spatial cues. 若干个bi-network串接起来, 形成一逐层上采样的网络结构.
  2. dense field之前没接触过, 可以了解一下

Applications

Face Swapping

Face Swapping

[Paper] On Face Segmentation, Face Swapping, and Face Perception
[Year] FG 2018
[Author] Yuval Nirkin, Iacopo Masi, Anh Tuan Tran, Tal Hassner, Gerard Medioni
[Pages] https://github.com/YuvalNirkin/face_swap
[Description]

3D

3DMM

[Paper] Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network
[Year] CVPR 2017
[Author] Anh Tuan Tran, Tal Hassner, Iacopo Masi, Iacopo Masi
[Pages]
https://www.openu.ac.il/home/hassner/projects/CNN3DMM/
https://github.com/anhttran/3dmm_cnn
[Description]

Datasets

Segments

Helen http://pages.cs.wisc.edu/~lizhang/projects/face-parsing/
LFW http://vis-www.cs.umass.edu/lfw/part_labels/
Mut1ny http://www.mut1ny.com/face-headsegmentation-dataset

Landmarks

LFW http://vis-www.cs.umass.edu/lfw/
Helen http://www.ifp.illinois.edu/~vuongle2/helen/
LFPW https://neerajkumar.org/databases/base/databases/lfpw/
WIDER http://mmlab.ie.cuhk.edu.hk/projects/WIDERFace/
http://www.face-rec.org/databases/
WFLW https://wywu.github.io/projects/LAB/WFLW.html (96 points)
ICME2019 https://facial-landmarks-localization-challenge.github.io/ (106 points)

Librarys

libfacedetection https://github.com/ShiqiYu/libfacedetection

Resources-Lists

https://blog.csdn.net/chenriwei2/article/details/50631212
https://github.com/betars/Face-Resources
https://blog.csdn.net/u011995719/article/details/78890333
https://www.jianshu.com/p/e4b9317a817f

About