GuideWsp / PedSurvey

From Handcrafted to Deep Features for Pedestrian Detection: A Survey

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

From Handcrafted to Deep Features for Pedestrian Detection: A Survey

  • This project provides a paper list about pedestrian detection following the taxonomy in our survey paper.
  • The performance of some methods on different datasets are shown in Comparison.md.
  • If you find a new paper about pedestrian detection, please feel free to contact us for adding it.

Table of Contents

  1. Detection pipeline
    1.1 Proposal generation
    1.2 Feature extraction
    1.3 Proposal classification
    1.4 Post processing
  2. Single-spectral pedestrian detection
    2.1 Handcrafted features based pedestrian detection
    2.1.1 Decision forests based methods
    2.1.2 Deformable part based methods
    2.2 Deep features based pedestrian detection
    2.2.1 Hybrid methods
    2.2.2 Pure CNN based methods
  3. Multispectral pedestrian detection
    3.1 Deep feature fusion
    3.2 Data processing
    3.3 Domain adaptation
  4. Datasets
    4.1 Earlier pedestrian datasets
    4.2 Modern pedestrian datasets
    4.3 Multispectral pedestrian datasets
  5. Challenges
    5.1 Scale variance
    5.2 Occlusion
    5.3 Domain adaptation
  6. Leaderboard

1. Detection pipeline

  • 1.1. Proposal generation
    • Sliding windows
    • Particle windows
      • Multistage particle windows for fast and accurate object detection, PAMI 2011. [Paper]
      • Learning sampling distributions for efficient object detection, TIP 2017. [Paper]
    • Objectness methods
      • Edge boxes: Locating object proposals from edges, ECCV 2014. [Paper]
      • Bing: Binarized normed gradients for objectness estimation at 300fps, CVPR 2014. [Paper]
      • What makes for effective detection proposals, PAMI 2016. [Paper]
      • Selective search for object recognition, IJCV 2016. [Paper]
    • Region proposal networks
      • Faster rcnn: Towards real-time object detection with region proposal networks, NIPS 2015. [Paper]
      • A unified multi-scale deep convolutional neural network for fast object detection, ECCV 2016. [Paper]
      • Region proposal by guided anchoring, CVPR 2019. [Paper]
  • 1.2. Feature extraction
    • Handcrafted features
      • Robust real-time face detection, IJCV 2004. [Paper]
      • Histograms of oriented gradients for human detection, CVPR 2005. [Paper]
      • Integral channel features, BMVC 2009. [Paper]
      • Object detection with discriminatively trained partbased models, CVPR 2009/PAMI 2010. [Paper]
    • Deep features
      • Imagenet classification with deep convolutional neural networks, NIPS 2012. [Paper]
      • Very deep convolutional networks for large-scale image recognition, arXiv 2014. [Paper]
      • Deep residual learning for image recognition, CVPR 2016. [Paper]
      • Densely connected convolutional networks, CVPR 2017. [Paper]
  • 1.3. Proposal classification/regression
    • Support-vector networks, ML 1995. [Paper]
    • A decision-theoretic generalization of on-line learning and an application to boosting, JCSS 1997. [Paper]
    • Softmax layer, Sigmoid layer, Smooth L1 layer
  • 1.4. Post processing
    • Greedy NMS
    • Soft-nms–improving object detection with one line of code, ICCV 2017. [Paper]
    • Learning nonmaximum suppression, CVPR 2017. [Paper]
    • Relation networks for object detection, CVPR 2018. [Paper]
    • Learning to separate: Detecting heavily-occluded objects in urban scenes, arXiv 2019. [Paper]
    • Adaptive nms: Refining pedestrian detection in a crowd, CVPR 2019. [Paper]

2. Single-spectral pedestrian detection

2.1. Handcrafted features based pedestrian detection

  • 2.1.1. Decision forests based methods
    • Robust real-time face detection, IJCV 2004. [Paper]
    • Integral channel features, BMVC 2009. [Paper]
    • New features and insights for pedestrian detection, CVPR 2010. [Paper]
    • Fast feature pyramids for object detection, BMVC 2010/PAMI 2014. [Paper]
    • Crosstalk cascades for frame-rate pedestrian detection, ECCV 2012. [Paper]
    • Seeking the strongest rigid detector, CVPR 2013. [Paper]
    • Exploring weak stabilization for motion feature extraction, CVPR 2013. [Paper]
    • Informed haar-like features improve pedestrian detection, CVPR 2014. [Paper]
    • Local decorrelation for improved pedestrian detection, NIPS 2014. [Paper]
    • Ten Years of Pedestrian Detection, What Have We Learned, ECCV 2014. [Paper]
    • Exploring human vision driven features for pedestrian detection, TCSVT 2015. [Paper]
    • Filtered channel features for pedestrian detection, CVPR 2015. [Paper]
    • Looking at pedestrians at different scales: A multiresolution approach and evaluations, TITS 2016. [Paper]
    • Semantic channels for fast pedestrian detection, CVPR 2016. [Paper]
    • How far are we from solving pedestrian detection? CVPR 2016. [Paper]
    • Pedestrian detection inspired by appearance constancy and shape symmetry, CVPR 2016/TIP 2016. [Paper]
    • Pedestrian detection with spatially pooled features and structured ensemble learning, ECCV 2016/PAMI 2017. [Paper]
    • Discriminative latent semantic feature learning for pedestrian detection, Neurocomputing 2017. [Paper]
    • A novel pixel neighborhood differential statistic feature for pedestrian and face detection, PR 2017. [Paper]
    • Local co-occurrence selection via partial least squares for pedestrian detection, TITS 2017. [Paper]
    • Fast boosting based detection using scale invariant multimodal multiresolution filtered features, CVPR 2017. [Paper]
    • Pedestrian detection by feature selected self-similarity features, IEEE Access 2018. [Paper]
    • An extended filtered channel framework for pedestrian detection, TITS 2018. [Paper]
    • Lbp channels for pedestrian detection, WACV 2018. [Paper]
    • Pedestrian proposal and refining based on the shared pixel differential feature, TITS 2019. [Paper]
    • Group cost-sensitive boostlr with vector form decorrelated filters for pedestrian detection, TITS 2019. [Paper]
    • Pedestrian detection using pixel difference matrix projection, TITS 2020. [paper]
  • 2.1.2. Deformable part based methods
    • Histograms of oriented gradients for human detection, CVPR 2005. [Paper]
    • Object detection with discriminatively trained partbased models, CVPR 2009/TPAMI 2010. [Paper]
    • Cascade object detection with deformable part models, CVPR 2010. [Paper]
    • Multiresolution models for object detection, ECCV 2010. [Paper]
    • Robust multi-resolution pedestrian detection in traffic scenes, CVPR 2013. [Paper]
    • Single-pedestrian detection aided by multi-pedestrian detection, CVPR 2013/TPAMI 2015. [Paper]
    • Regionlets for generic object detection, CVPR 2013/TPAMI 2015. [Paper]
    • Pedestrian detection in crowded scenes via scale and occlusion analysis, ICIP 2016. [paper]
    • Real-time rgb-d based template matching pedestrian detection, ICRA 2016. [paper]
    • A pedestrian detection system accelerated by kernelized proposals, TITS 2020. [paper]

2.2. Deep features based pedestrian detection

  • Hybrid methods

    • 2.2.1. CNN as feature
      • Convolutional channel features, ICCV 2015. [Paper]
      • Learning complexity-aware cascades for deep pedestrian detection, ICCV 2015. [Paper]
      • Is faster r-cnn doing well for pedestrian detection? ECCV 2016. [Paper]
      • Learning multilayer channel features for pedestrian detection, TIP 2017. [Paper]
      • Neural features for pedestrian detection, Neurocomputing 2017. [Paper]
      • Filtered shallow-deep feature channels for pedestrian detection, Neurocomputing 2017. [Paper]
      • Pushing the limits of deep cnns for pedestrian detection, TCSVT 2018. [Paper]
      • Fast pedestrian detection with attention-enhanced multi-scale rpn and soft-cascaded decision trees, TITS 2019. [paper]
      • Hybrid channel based pedestrian detection, Neurocomputing 2020. [Paper]
    • CNN as classifier
      • Joint deep learning for pedestrian detection, ICCV 2013. [Paper]
      • Switchable deep network for pedestrian detection, CVPR 2014. [Paper]
      • Taking a deeper look at pedestrians, CVPR 2015. [Paper]
      • Pedestrian detection aided by deep learning semantic tasks, CVPR 2015. [Paper]
      • Real-time pedestrian detection with deep network cascades, BMVC 2015. [Paper]
      • Deep learning strong parts for pedestrian detection, ICCV 2015. [Paper]
      • Deep network aided by guiding network for pedestrian detection, PRL 2017. [Paper]
      • Improving the performance of pedestrian detectors using convolutional learning, PR 2017. [Paper]
      • Scale-aware fast r-cnn for pedestrian detection, TMM 2018. [Paper]
      • Deepid-net: Object detection with deformable part based convolutional neural networks, TPAMI 2017. [Paper]
      • S-cnn: Subcategory-aware convolutional networks for object detection, TPAMI 2018. [Paper]
      • Jointly learning deep features, deformable parts, occlusion and classification for pedestrian detection, TPAMI 2018. [Paper]
  • 2.2.2. Pure CNN based methods

    • Scale-aware methods
      • Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers, CVPR 2016. [Paper]
      • A unified multi-scale deep convolutional neural network for fast object detection, ECCV 2016. [Paper]
      • Scale-adaptive deconvolutional regression network for pedestrian detection, ACCV 2016. [Paper]
      • Sam-rcnn: Scaleaware multi-resolution multi-channel pedestrian detection, BMVC 2018. [Paper]
      • Fpn++: A simple baseline for pedestrian detection, ICME 2019. [Paper]
    • Part-based methods
      • PCN: Part and context information for pedestrian detection with cnns, BMVC 2017. [Paper]
      • Joint holistic and partial cnn for pedestrian detection, BMVC 2018. [Paper]
      • Occlusion-aware r-cnn: Detecting pedestrians in a crowd, ECCV 2018. [Paper]
      • Bi-box regression for pedestrian detection and occlusion estimation, ECCV 2018. [Paper]
      • Pedjointnet: Joint headshoulder and full body deep network for pedestrian detection, IEEE Access 2019. [Paper]
      • Double anchor r-cnn for human detection in a crowd, arXiv 2019. [Paper]
      • Semantic head enhanced pedestrian detection in a crowd, arXiv 2019. [Paper]
      • Semantic part rcnn for real-world pedestrian detection, CVPRW 2019. [Paper]
      • Mask-guided attention network for occluded pedestrian detection, ICCV 2019. [Paper]
    • Attention-based methods
      • Illuminating pedestrians via simultaneous detection and segmentation, ICCV 2017. [Paper]
      • Vis-hud: Using visual saliency to improve human detection with convolutional neural networks, CVPRW 2018. [Paper]
      • Graininess-aware deep feature learning for pedestrian detection, ECCV 2018. [Paper]
      • Occluded pedestrian detection through guided attention in cnns, CVPR 2018. [Paper]
      • Deep feature fusion by competitive attention for pedestrian detection, IEEE Access 2019. [Paper]
      • Part-level convolutional neural networks for pedestrian detection using saliency and boundary box alignment, IEEE Access 2019. [Paper]
      • Multi-grained deep feature learning for robust pedestrian detection, TCSVT 2019. [Paper]
      • Attention guided neural network models for occluded pedestrian detection, PR 2020. [Paper]
    • Feature-fused methods
      • Direct multi-scale dual-stream network for pedestrian detection, ICIP 2017. [Paper]
      • Accurate single stage detector using recurrent rolling convolution, CVPR 2017. [Paper]
      • Object detection based on multilayer convolution feature fusion and online hard example mining, IEEE Access 2018. [Paper]
      • Pedestrian detection via body part semantic and contextual information with dnn, TMM 2018. [Paper]
      • Deep aggregation learning for high-performance small pedestrian detection, ACML 2018. [Paper]
      • Learning pixel-level and instance-level context-aware features for pedestrian detection in crowds, IEEE Access 2019. [Paper]
      • Mfr-cnn: Incorporating multi-scale features and global information for traffic object detection, TVT 2019. [Paper]
      • Taking a look at small-scale pedestrians and occluded pedestrians, TIP 2019. [Paper]
      • Coupled network for robust pedestrian detection with gated multi-layer feature extraction and deformable occlusion handling Cascade-based methods, arXiv 2019. [Paper]
      • Object detection with location-aware deformable convolution and backward attention filtering, CVPR 2019. [Paper]
      • Temporal-context enhanced detection of heavily occluded pedestrians, CVPR 2020. [Paper]
    • Cascade-based methods
      • Fused dnn: A deep neural network fusion approach to fast and robust pedestrian detection, WACV 2017. [Paper]
      • Learning efficient single-stage pedestrian detectors by asymptotic localization fitting, ECCV 2018. [Paper]
      • Circlenet: Reciprocating feature adaptation for robust pedestrian detection, TITS 2019. [Paper]
      • Pedestrian detection with autoregressive network phases, CVPR 2019. [Paper]
      • Pedestrian detection: The elephant in the room, arXiv 2020. [Paper]
      • A one-and-half stage pedestrian detector, WACV 2020. [Paper]
      • Progressive Refinement Network for Occluded Pedestrian Detection, ECCV 2020. [Paper]
    • Anchor-free methods
      • Small-scale pedestrian detection based on topological line localization and temporal feature aggregation, ECCV 2018. [Paper]
      • High-level semantic feature detection: A new perspective for pedestrian detection, CVPR 2019. [Paper]
      • Attribute-aware pedestrian detection in a crowd, arXiv 2019. [Paper]
    • Data-augmentation methods
      • Synthesizing a scene-specific pedestrian detector and pose estimator for static video surveillance, IJCV 2018. [Paper]
      • Training cascade compact cnn with region-iou for accurate pedestrian detection, TITS 2019. [Paper]
      • A shape transformation-based dataset augmentation framework for pedestrian detection, arXiv 2019. [Paper]
      • Advanced pedestrian dataset augmentation for autonomous driving, ICCVW 2019. [Paper]
      • Pmc-gans: Generating multi-scale high-quality pedestrian with multimodal cascaded gans, BMVC 2019. [Paper]
      • Pedhunter: Occlusion robust pedestrian detector in crowded scenes, AAAI 2020. [Paper]
      • Where, what, whether: Multi-modal learning meets pedestrian detection, CVPR 2020. [Paper]
    • Loss-driven methods
      • Perceptual generative adversarial networks for small object detection, CVPR 2017. [Paper]
      • Mimicking very efficient network for object detection, CVPR 2017. [Paper]
      • Fused discriminative metric learning for low resolution pedestrian detection, ICIP 2017. [Paper]
      • Boosted convolutional neural networks (bcnn) for pedestrian detection, WACV 2017. [Paper]
      • Subcategory-aware convolutional neural networks for object proposals and detection, WACV 2017. [Paper]
      • Repulsion loss: Detecting pedestrians in a crowd, CVPR 2018. [Paper]
      • Learning lightweight pedestrian detector with hierarchical knowledge distillation, ICIP 2019. [Paper]
      • Discriminative feature transformation for occluded pedestrian detection, ICCV 2019. [Paper]
      • Count- and Similarity-aware R-CNN for Pedestrian Detection, ECCV 2020. [Paper]
    • Post-processing methods
      • End-to-end people detection in crowded scenes, CVPR 2016. [Paper]
      • Led: Localization-quality estimation embedded detector, ICIP 2018. [Paper]
      • Learning to separate: Detecting heavily-occluded objects in urban scenes, arXiv 2019. [Paper]
      • Single shot multibox detector with kalman filter for online pedestrian detection in video, IEEE Access 2019. [Paper]
      • Adaptive nms: Refining pedestrian detection in a crowd, CVPR 2019. [Paper]
      • S3d: Scalable pedestrian detection via score scale surface discrimination, TCSVT 2020. [Paper]
      • Nms by representative region: Towards crowded pedestrian detection by proposal pairing, CVPR 2020. [Paper]
    • Multi-task methods
      • What can help pedestrian detection? CVPR 2017. [Paper]
      • Accurate pedestrian detection by human pose regression, TIP 2019. [Paper]
      • Human detection aided by deeply learned semantic masks, TCSVT 2019. [Paper]
      • Cluenet: A deep framework for occluded pedestrian pose estimation, BMVC 2019. [Paper]
      • Semantic part rcnn for real-world pedestrian detection, CVPRW 2019. [Paper]
      • Re-id driven localization refinement for person search, ICCV 2019. [Paper]
    • others
      • Exploiting target data to learn deep convolutional networks for scene-adapted human detection, TIP 2018. [Paper]
      • Deep learning approaches on pedestrian detection in hazy weather, TIE 2019. [Paper]
      • Pedestrian detection from thermal images using saliency maps, CVPRW 2019. [Paper]
      • Domainadaptive pedestrian detection in thermal images, ICIP 2019. [Paper]
      • Spatial focal loss for pedestrian detection in fisheye imagery, WACV 2019.[Paper]
      • Oriented spatial transformer network for pedestrian detection using fish-eye camera, TMM 2020. [Paper]
      • Semi-supervised human detection via region proposal networks aided by verification, TIP 2020. [Paper]
      • Task-conditioned Domain Adaptation for Pedestrian Detection in Thermal Imagery, ECCV 2020. [Paper]

3. Multispectral pedestrian detection

3.1. Deep feature fusion

  • Multispectral deep neural networks for pedestrian detection, BMVC 2016. [Paper]
  • Fully convolutional region proposal networks for multispectral person detection, CVPR 2017. [Paper]
  • Pedestrian detection for autonomous vehicle using multi-spectral cameras, TIV 2019. [Paper]
  • Fusion of multispectral data through illuminationaware deep neural networks for pedestrian detection, IF 2019. [Paper]
  • Illuminationaware faster r-cnn for robust multispectral pedestrian detection, PR 2019. [Paper]
  • Cross-modality interactive attention network for multispectral pedestrian detection, IF 2019. [Paper]
  • Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems, ECCV 2020. [Paper]

3.2. Data processing

  • Multispectral pedestrian detection via simultaneous detection and segmentation, BMVC 2018. [Paper]
  • Weakly aligned cross-modal learning for multispectral pedestrian detection, ICCV 2019. [Paper]

3.3. Domain adaptation

  • Learning cross-modal deep representations for robust pedestrian detection, CVPR 2017. [Paper]
  • Unsupervised domain adaptation for multispectral pedestrian detection, CVPRW 2019. [Paper]
  • Pedestrian detection with unsupervised multispectral feature learning using deep neural networks, IF 2019. [Paper]

4. Datasets

4.1. Earlier pedestrian datasets

  • A trainable system for object detection, IJCV 2000. [Paper]
  • Histograms of oriented gradients for human detection, CVPR 2005. [Paper]
  • Depth and appearance for mobile scene analysis, ICCV 2007. [Paper]
  • Multi-cue onboard pedestrian detection, CVPR 2009. [Paper]
  • Monocular pedestrian detection: Survey and experiments, TPAMI 2009. [Paper]

4.2. Modern pedestrian datasets

  • Pedestrian detection: An evaluation of the state of the art, PAMI 2010. [Paper]
  • Are we ready for autonomous driving? the kitti vision benchmark suite, CVPR 2012. [Paper]
  • Citypersons: A diverse dataset for pedestrian detection, CVPR 2017. [Paper]
  • Nightowls: A pedestrians at night dataset, ACCV 2018. [Paper]
  • Crowdhuman: A benchmark for detecting human in a crowd, arXiv 2018. [Paper]
  • Eurocity persons: A novel benchmark for person detection in traffic scenes, PAMI 2019. [Paper]
  • Widerperson: A diverse dataset for dense pedestrian detection in the wild, TMM 2020. [Paper]

4.3. Multispectral pedestrian datasets

  • Multispectral pedestrian detection: Benchmark dataset and baseline, CVPR 2015. [Paper]
  • Pedestrian detection at day/night time with visible and fir cameras: A comparison, PR 2016. [Paper]

5. Challenges

5.1. Scale variance

  • A unified multi-scale deep convolutional neural network for fast object detection, ECCV 2016. [Paper]
  • Exploit all the layers: Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers, CVPR 2016. [Paper]
  • Feature pyramid networks for object detection, CVPR 2017. [Paper]
  • Perceptual generative adversarial networks for small object detection, CVPR 2017. [Paper]
  • Task-driven super resolution: Object detection in low-resolution images, arXiv 2018. [Paper]
  • High-level semantic networks for multi-scale object detection, TCSVT 2019. [Paper]
  • Small-scale pedestrian detection based on deep neural network, TITS 2019. [Paper]
  • Scale-aware fast r-cnn for pedestrian detection, TMM 2019. [Paper]
  • Jcs-net: Joint classification and super-resolution network for small-scale pedestrian detection in surveillance images, TIFS 2019. [Paper]
  • Multi-resolution generative adversarial networks for tinyscale pedestrian detection, ICIP 2019. [Paper]
  • Scale match for tiny person detection, WACV 2020. [Paper]
  • Self-Mimic Learning for Small-scale Pedestrian Detection, ACM-MM 2020. [Paper]

5.2. Occlusion

  • An hog-lbp human detector with partial occlusion handling, ICCV 2010. [Paper]
  • Handling occlusions with franken-classifiers, CVPR 2013. [Paper]
  • Deep learning strong parts for pedestrian detection, ICCV 2015. [Paper]
  • Multi-label learning of part detectors for heavily occluded pedestrian detection, ICCV 2017. [Paper]
  • Repulsion loss: Detecting pedestrians in a crowd, CVPR 2018. [Paper]
  • Improving occlusion and hard negative handling for single-stage pedestrian detectors, CVPR 2018. [Paper]
  • Bi-box regression for pedestrian detection and occlusion estimation, ECCV 2018. [Paper]
  • Occlusion-aware r-cnn: Detecting pedestrians in a crowd, ECCV 2018. [Paper]
  • Adaptive nms: Refining pedestrian detection in a crowd, CVPR 2019. [Paper]
  • Mask-guided attention network for occluded pedestrian detection, ICCV 2019. [Paper]
  • Learning to separate: Detecting heavily-occluded objects in urban scenes, arXiv 2019. [Paper]
  • Psc-net: Learning part spatial cooccurence for occluded pedestrian detection, arXiv 2020. [Paper]
  • Detection in crowded scenes: One proposal, multiple predictions, CVPR 2020. [Paper]
  • Pedhunter: Occlusion robust pedestrian detector in crowded scenes, AAAI 2020. [Paper]
  • Relational learning for joint head and human detection, AAAI 2020. [Paper]

5.3. Domain adaptation

  • Domain adaptive faster r-cnn for object detection in the wild, CVPR 2018. [Paper]
  • Progressive domain adaptation for object detection, CVPRW 2018. [Paper]
  • A robust learning approach to domain adaptive object detection, ICCV 2019. [Paper]
  • Diversify and match: A domain adaptive representation learning paradigm for object detection, CVPR 2019. [Paper]
  • Domain adaptation for object detection via style consistency, BMVC 2019. [Paper]
  • Strong-weak distribution alignment for adaptive object detection, CVPR 2019. [Paper]
  • Few-shot adaptive faster r-cnn, CVPR 2019. [Paper]
  • Multi-level domain adaptive learning for cross-domain detection, ICCVW 2019. [Paper]
  • Adapting object detectors via selective cross-domain alignment, CVPR 2019. [Paper]

Citation

If this project help your research, please consider to cite our survey paper.

@article{Cao_PDR_arXiv_2020,
         author = {Jiale Cao and Yanwei Pang and Jin Xie and Fahad Shahbaz Khan and Ling Shao},
         title = "{From Handcrafted to Deep Features for Pedestrian Detection: A Survey}",
         journal = {arXiv},
         year = "2020"
        }

Contact

Please contact us if you have any questions.

About

From Handcrafted to Deep Features for Pedestrian Detection: A Survey