End-to-End People Detection in Crowded Scenes
Abstract
Current people detectors operate either by scanning an image in a sliding window fashion or by classifying a discrete set of proposals. We propose a model that is based on decoding an image into a set of people detections. Our system takes an image as input and directly outputs a set of distinct detection hypotheses. Because we generate predictions jointly, common post-processing steps such as nonmaximum suppression are unnecessary. We use a recurrent LSTM layer for sequence generation and train our model end-to-end with a new loss function that operates on sets of detections. We demonstrate the effectiveness of our approach on the challenging task of detecting people in crowded scenes1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Using FPN as head detector, performance on Brainwash is much better than state-of-the-art in [23].
“By using the FPN as the head detector, the performance is already much better than the state-ofart in [23].”
From CrowdHuman: A Benchmark for Detecting Human in a Crowd (2018) · cites this paper as [23]
Closest to DETR are end-to-end set predictions for object detection and instance segmentation.
“Closest to our approach are end-to-end set predictions for object detection [43] and instance segmentation [41,30,36,42].”
From End-to-End Object Detection with Transformers (2020) · cites this paper as [42]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
The Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · see citing works
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Going Deeper with Convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014) · arXiv · doi:10.48550/arxiv.1409.4842 · Cited by 1,461 · see citing works
Learning To Count Objects in Images
Lempitsky, V., & Zisserman, A. (2010) · Cited by 1,063 · see citing works
Pedestrian Detection in Crowded Scenes
Leibe, B., Seemann, E., & Schiele, B. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.272 · Cited by 871 · see citing works
People-tracking-by-detection and people-detection-by-tracking
Andriluka, M., Roth, S., & Schiele, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587583 · Cited by 825 · see citing works
Self-citationAn Empirical Evaluation of Deep Learning on Highway Driving
Huval, B., Wang, T., Tandon, S., Kiske, J., Song, W., Pazhayampallil, J., Andriluka, M., Rajpurkar, P., Migimatsu, T., Cheng-Yue, R., et al. (2015) · arXiv · doi:10.48550/arxiv.1504.01716 · Cited by 537 · see citing works
Self-citationRecognition using visual phrases
Sadeghi, M. A., & Farhadi, A. (2011) · doi:10.1109/cvpr.2011.5995711 · Cited by 421 · see citing works
Filtered channel features for pedestrian detection
Zhang, S., Benenson, R., & Schiele, B. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298784 · Cited by 395 · see citing works
Multi-cue onboard pedestrian detection
Wojek, C., Walk, S., & Schiele, B. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206638 · Cited by 350 · see citing works
Scalable, High-Quality Object Detection
Szegedy, C., Reed, S., Erhan, D., Anguelov, D., & Ioffe, S. (2014) · arXiv · doi:10.48550/arxiv.1412.1441 · Cited by 333 · see citing works
On Detection of Multiple Object Instances Using Hough Transforms
Barinova, O., Lempitsky, V., & Kholi, P. K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.79 · Cited by 237 · see citing works
Single-Pedestrian Detection Aided by Multi-pedestrian Detection
Ouyang, W., & Wang, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.411 · Cited by 144 · see citing works
On detection of multiple object instances using hough transforms
Barinova, O., Lempitsky, V., & Kohli, P. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539905 · Cited by 125 · see citing works
Learning People Detectors for Tracking in Crowded Scenes
Tang, S., Andriluka, M., Milan, A., et al. (2013) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.134 · Cited by 97 · see citing works
Self-citationWan, L., Eigen, D., & Fergus, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298686 · Cited by 86 · see citing works
Detection and Tracking of Occluded People
Tang, S., Andriluka, M., & Schiele, B. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.9 · Cited by 53 · see citing works
Self-citationEnd-to-end people detection in crowded scenesPreprint
Stewart, R., & Andriluka, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04878 · Cited by 33 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · see citing works
Relation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524 · see citing works
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · see citing works
“TrackFormer extends the recently proposed set prediction objective for object detection [7, 47, 68] to multi-object tracking.”
BackgroundMethodLearning Non-maximum Suppression
Hosang, J., Benenson, R., & Schiele, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.685 · Cited by 736 · see citing works
MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving
Teichmann, M., Weber, M., Zollner, M., Cipolla, R., & Urtasun, R. (2018) · 2018 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2018.8500504 · Cited by 681 · see citing works
Deep learning in video multi-object tracking: A survey
Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., & Herrera, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.023 · Cited by 665 · see citing works
CrowdHuman: A Benchmark for Detecting Human in a Crowd
Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., & Sun, J. (2018) · arXiv · doi:10.48550/arxiv.1805.00123 · Cited by 534 · see citing works
“and then finetune it on the visible body detection benchmarks like COCOPersons [17], full body detection benchmarks like Caltech [6] and CityPersons [31], and head detection benchmarks like Brainwash [23]. As reported in Section4.4, FPN is superior to RetinaNet in all three cases. Therefore, in the following experiment…”
BackgroundMethodInfluentialAdaptive NMS: Refining Pedestrian Detection in a Crowd
Liu, S., Huang, D., & Wang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00662 · Cited by 413 · see citing works
SINet: A Scale-Insensitive Convolutional Neural Network for Fast Vehicle Detection
Hu, X., Xu, X., Xiao, Y., Chen, H., He, S., Qin, J., & Heng, P.-A. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2018.2838132 · Cited by 341 · see citing works
End-to-End Instance Segmentation with Recurrent Attention
Ren, M., & Zemel, R. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.39 · Cited by 309 · see citing works
Pose Recognition with Cascade Transformers
Li, K., Wang, S., Zhang, X., Xu, Y., Xu, W., & Tu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00198 · Cited by 301 · see citing works
End-to-End Object Detection with Fully Convolutional Network
Wang, J., Song, L., Li, Z., Sun, H., Sun, J., & Zheng, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01559 · Cited by 235 · see citing works
“Concretely, in the earlier researches, numerous detection frameworks based on recurrent neural networks [43, 35, 26, 32, 36] attempt to produce a set of bounding boxes directly.”
BackgroundMethodAnchor Box Optimization for Object Detection
Zhong, Y., Wang, J., Peng, J., & Zhang, L. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093498 · Cited by 118 · see citing works
A comprehensive review of recent advances on deep vision systems
Abbas, Q., Ibrahim, M. E. A., & Jaffar, M. A. (2019) · Artificial Intelligence Review · doi:10.1007/s10462-018-9633-3 · Cited by 99 · see citing works
Object Detection with Transformers: A Review
Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., & Afzal, M. Z. (2025) · Sensors · doi:10.3390/s25196025 · Cited by 93 · see citing works
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84 · see citing works
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Wang, Y., Guizilini, V., Zhang, T., Wang, Y., Zhao, H., & Solomon, J. (2021) · arXiv · doi:10.48550/arxiv.2110.06922 · Cited by 182 · see citing works
“Following [30, 10], we use a set-to-set loss to measure the discrepancy between the prediction set ( ˆ B (cid:96) , ˆ C (cid:96) ) and the ground-truth set ( B , C ) .”
MethodInfluentialRecurrent Neural Networks for Semantic Instance Segmentation
Salvador, A., Bellver, M., Campos, V., Baradad, M., Marques, F., Torres, J., & Giro-i-Nieto, X. (2017) · arXiv · doi:10.48550/arxiv.1712.00617 · Cited by 62 · see citing works
Revisiting Monocular Satellite Pose Estimation With Transformer
Wang, Z., Zhang, Z., Sun, X., Li, Z., & Yu, Q. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3161605 · Cited by 59 · see citing works
DeepSetNet: Predicting Sets with Deep Neural Networks
Rezatofighi, S. H., B. G., V. K., Milan, A., Abbasnejad, E., Dick, A., & Reid, I. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.561 · Cited by 45 · see citing works
Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
Peng, K., Mathur, A., & Narayanan, A. (2021) · arXiv · doi:10.48550/arxiv.2108.02922 · Cited by 31 · see citing works
Zhang, F., Yang, F., Li, C., & Yuan, G. (2019) · IEEE Access · doi:10.1109/access.2019.2919103 · Cited by 29 · see citing works
Seidel, R., Jahn, N., Seo, S., Goerttler, T., & Obermayer, K. (2022) · IEEE Open Journal of Intelligent Transportation Systems · doi:10.1109/ojits.2021.3139393 · Cited by 28 · see citing works
- Authors
- Stewart, R., Andriluka, M., & Ng, A. Y.
- Affiliations
- Stanford University (US)
- Max Planck Institute for Informatics (DE)
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 2325-2333
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W607748843
- Cited by
- 539
- References
- 29
- Citations by year
- 2026: 8 · 2025: 42 · 2024: 64 · 2023: 66 · 2022: 57 · 2021: 91 · 2020: 46 · 2019: 66 · 2018: 44 · 2017: 37 · 2016: 17 · 2015: 1
- FWCI
- 18.97
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human detection, crowded scenes, end-to-end detection, sequence generation, LSTM, joint detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
