From Handcrafted to Deep Features for Pedestrian Detection: A Survey
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
Pedestrian detection is an important but challenging problem in computer vision, especially in human-centric tasks. Over the past decade, significant improvement has been witnessed with the help of handcrafted features and deep features. Here we present a comprehensive survey on recent advances in pedestrian detection. First, we provide a detailed review of single-spectral pedestrian detection that includes handcrafted features based methods and deep features based approaches. For handcrafted features based methods, we present an extensive review of approaches and find that handcrafted features with large freedom degrees in shape and space have better performance. In the case of deep features based approaches, we split them into pure CNN based methods and those employing both handcrafted and CNN based features. We give the statistical analysis and tendency of these methods, where feature enhanced, part-aware, and post-processing methods have attracted main attention. In addition to single-spectral pedestrian detection, we also review multi-spectral pedestrian detection, which provides more robust features for illumination variance. Furthermore, we introduce some related datasets and evaluation metrics, and a deep experimental analysis. We conclude this survey by emphasizing open problems that need to be addressed and highlighting various future directions. Researchers can track an up-to-date list at https://github.com/JialeCao001/PedSurvey.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398
SURF: Speeded Up Robust Features
Bay, H., Tuytelaars, T., & Van Gool, L. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_32 · Cited by 14,955
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · cited 1×
“, pedestrian detection [6], [43], [242], person re-identification [85], [104], [226], [253], person search [65], [135], [211], [219], pose estimation [21], [133], [137], [166], and face detection [100], [130], [161], [224]) have gained significant attention over the past decade.”
BackgroundCascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · cited 1×
“, pedestrian detection [6], [43], [242], person re-identification [85], [104], [226], [253], person search [65], [135], [211], [219], pose estimation [21], [133], [137], [166], and face detection [100], [130], [161], [224]) have gained significant attention over the past decade.”
BackgroundDeep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
Edge Boxes: Locating Object Proposals from Edges
Zitnick, C. L., & Dollár, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_26 · Cited by 2,656
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274 · cited 3×
“[202] proposed to use soft-NMS [7] to select some occluded samples for training.”
BackgroundDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084
Single-Shot Refinement Neural Network for Object Detection
Zhang, S., Wen, L., Bian, X., Lei, Z., & Li, S. Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00442 · Cited by 1,678
Domain Adaptive Faster R-CNN for Object Detection in the Wild
Chen, Y., Li, W., Sakaridis, C., Dai, D., & Van Gool, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00352 · Cited by 1,672
A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection
Cai, Z., Fan, Q., Feris, R. S., & Vasconcelos, N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_22 · Cited by 1,605 · cited 6×
“Based on MS-CNN [11], Jung et al. [84] further combined the features of consecutive layers.”
BackgroundMethodInfluentialRelation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524
A Trainable System for Object Detection
Papageorgiou, C., & Poggio, T. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008162616689 · Cited by 1,289 · cited 2×
“MIT [152] IJCV2000 924 64×128 full day one of earliest pedestrian datasets”
BackgroundMultispectral pedestrian detection: Benchmark dataset and baseline
Hwang, S., Park, J., Kim, N., Choi, Y., & Kweon, I. S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298706 · Cited by 1,249
Dollar, P., Tu, Z., Perona, P., & Belongie, S. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.91 · Cited by 1,237
Monocular Pedestrian Detection: Survey and Experiments
Enzweiler, M., & Gavrila, D. M. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.260 · Cited by 1,213 · cited 2×
“Daimler [45] PAMI2009 29k 72k 640×480 full day gray-color images in urban traffic”
BackgroundThe Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174
Perceptual Generative Adversarial Networks for Small Object Detection
Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., & Yan, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.211 · Cited by 953 · cited 3×
“Scale-aware Part-aware Context Description DDN [138] CVPR2012 Hybrid SW HOG softmax NMS no yes no first deep model for pedestrian detection UMS [169] CVPR2013 P-CNN SW CNN softmax NMS no no ms fusion one of the earliest deep pedestrian detectors UDN [139] ICCV2013 P-CNN SW CNN softmax NMS no yes no join different comp…”
BackgroundRobust real-time face detection
Viola, P., & Jones, M. (2005) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937709 · Cited by 898
Is Faster R-CNN Doing Well for Pedestrian Detection?
Zhang, L., Lin, L., Liang, X., & He, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_28 · Cited by 852
Strong-Weak Distribution Alignment for Adaptive Object Detection
Saito, K., Ushiku, Y., Harada, T., & Saenko, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00712 · Cited by 819 · cited 1×
“Most domain adaptation methods are based on adversarial learning, including data-level [72], [87], [89], [121], feature-level [168], [215], and instance-level [24], [264] methods.”
MethodLearning Non-maximum Suppression
Hosang, J., Benenson, R., & Schiele, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.685 · Cited by 736
What Makes for Effective Detection Proposals?
Hosang, J., Benenson, R., Dollar, P., & Schiele, B. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2465908 · Cited by 727
Yang, F., Choi, W., & Lin, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.234 · Cited by 608 · cited 1×
“In contrast, feature pyramid strategy is widely adopted by deep features based methods [11], [20], [66], [101], [111], [221], which use the different layers of CNNs to detect objects of different scales.”
BackgroundEnd-to-End People Detection in Crowded Scenes
Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · cited 3×
“EEPD [179] CVPR2016 P-CNN - CNN LSTM no no no no end-to-end approach directly predicting objects”
BackgroundCrowdHuman: A Benchmark for Detecting Human in a Crowd
Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., & Sun, J. (2018) · arXiv · doi:10.48550/arxiv.1805.00123 · Cited by 534 · cited 7×
“The recently proposed Jaccard index (JI) represents the overlap between detection results and ground-truths [171], which is used to evaluate the performance in crowded scenes.”
BackgroundMethodInfluentialOcclusion-Aware R-CNN: Detecting Pedestrians in a Crowd
Zhang, S., Wen, L., Bian, X., Lei, Z., & Li, S. Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_39 · Cited by 488
Adaptive NMS: Refining Pedestrian Detection in a Crowd
Liu, S., Huang, D., & Wang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00662 · Cited by 413 · cited 24×
“Impact of post-processing Third, we show the impact of two post-processing strategies (i.e., NMS and SoftNMS) in Table 15.”
BackgroundMethodInfluentialFiltered channel features for pedestrian detection
Zhang, S., Benenson, R., & Schiele, B. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298784 · Cited by 395 · cited 14×
“Among these these methods, only four methods (i.e., ACF [39], Checkerboards [243], NNNF [17], and Regionlets [200], [201]) are handcrafted features based approaches and the remaining methods are deep features based approaches.”
BackgroundMethodInfluentialAdapting Object Detectors via Selective Cross-Domain Alignment
Zhu, X., Pang, J., Yang, C., Shi, J., & Lin, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00078 · Cited by 387
Progressive Domain Adaptation for Object Detection
Hsu, H.-K., Yao, C.-H., Tsai, Y.-H., Hung, W.-C., Tseng, H.-Y., Singh, M., & Yang, M.-H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093358 · Cited by 382 · cited 1×
“Most domain adaptation methods are based on adversarial learning, including data-level [72], [87], [89], [121], feature-level [168], [215], and instance-level [24], [264] methods.”
MethodDiversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection
Kim, T., Jeong, M., Kim, S., Choi, S., & Kim, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01274 · Cited by 352
Multi-cue onboard pedestrian detection
Wojek, C., Walk, S., & Schiele, B. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206638 · Cited by 350 · cited 2×
“TUD-Brussels [203] is recorded from a driving car in the city of Brussels and contains 508 images at a fixed resolution of 650×480 pixels.”
BackgroundGuan, D., Cao, Y., Yang, J., Cao, Y., & Yang, M. Y. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.11.017 · Cited by 342
Learning Complexity-Aware Cascades for Deep Pedestrian Detection
Cai, Z., Saberian, M., & Vasconcelos, N. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.384 · Cited by 326
EuroCity Persons: A Novel Benchmark for Person Detection in Traffic Scenes
Braun, M., Krebs, S., Flohr, F., & Gavrila, D. M. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2897684 · Cited by 307 · cited 2×
“EuroCity persons5 [8] is a large-scale dataset of urban scenes captured at both daytime and nighttime in multiple European cites.”
BackgroundWhat Can Help Pedestrian Detection?
Mao, J., Xiao, T., Jiang, Y., & Cao, Z. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.639 · Cited by 306
Accurate Single Stage Detector Using Recurrent Rolling Convolution
Ren, J., Chen, X., Liu, J., Sun, W., Pang, J., Yan, Q., Tai, Y.-W., & Xu, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.87 · Cited by 305
A discriminative deep model for pedestrian detection with occlusion handling
Ouyang, W., & Wang, X. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248062 · Cited by 281 · cited 3×
“features to pedestrian detection [138], [139], [169].”
BackgroundSingle-Pedestrian Detection Aided by Multi-pedestrian Detection
Ouyang, W., & Wang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.411 · Cited by 144
Regionlets for Generic Object Detection
Wang, X., Yang, M., Zhu, S., & Lin, Y. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389830 · Cited by 107
Pedestrian detection in automotive safety: Understanding state-of-the-art
Ragesh, N. K., & Rajesh, R. (2019) · IEEE Access · doi:10.1109/access.2019.2909992 · Cited by 57 · cited 1×
“Some surveys about pedestrian detection [6], [54], [159]”
Background
Cited by
Towards Large-Scale Small Object Detection: Survey and Benchmarks
Cheng, G., Yuan, X., Yao, X., Yan, K., Zeng, Q., Xie, X., & Han, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3290594 · Cited by 683
“Quite a number of surveys about object detection have been published in recent years [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], and our review differs from the existing ones mainly in two aspects.”
BackgroundFederated Transfer–Ordered–Personalized Learning for Driver Monitoring Application
Yuan, L., Su, L., & Wang, Z. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3279273 · Cited by 40
“, enabling pedestrian detection [2], automated driving [3], mobility digital twins [4], and other transportation applications.”
Background
Metadata
- Authors
- Cao, J., Pang, Y., Xie, J., Khan, F. S., & Shao, L.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-09-01
- Volume
- 44
- Issue
- 9
- Pages
- 4913-4934
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3090757175
- Cited by
- 160
- References
- 61
- FWCI
- 9.70
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.