A Survey of Multiple Pedestrian Tracking Based on Tracking-by-Detection Framework
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Multiple pedestrian tracking (MPT) has gained significant attention due to its huge potential in a commercial application. It aims to predict multiple pedestrian trajectories and maintain their identities, given a video sequence. In the past decade, due to the advancement in pedestrian detection algorithms, Tracking-by-Detection (TBD) based algorithms have achieved tremendous successes. TBD has become the most popular MPT framework, and it has been actively studied in the past decade. In this paper, we give a comprehensive survey of recent advances in TBD-based MPT algorithms. We systematically analyze the existing TBD-based algorithms and organize the survey into four major parts. At first, this survey draws a timeline to introduce the milestones of TBD-based works which briefly reviews the development of the existing TBD-based methods. Second, the main procedures of the TBD framework are summarized, and each stage in the procedure is described in detail. Afterward, this survey analyzes the performance of existing TBD-based algorithms on MOT challenge datasets and discusses the factors that affect tracking performance. Finally, open issues and future directions in the TBD framework are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853
Cascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370
Simple online and realtime tracking with a deep association metric
Wojke, N., Bewley, A., & Paulus, D. (2017) · 2017 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2017.8296962 · Cited by 5,286
Simple online and realtime tracking
Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. (2016) · 2016 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2016.7533003 · Cited by 4,433
Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics
Bernardin, K., & Stiefelhagen, R. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/246309 · Cited by 2,918
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274 · cited 1×
“The NMS algorithm, an integral part of many detection algorithms in computer vision [115], selects a bounding box with the maximum detection score and uses a pre-defined overlap threshold to suppress its neighboring boxes as shown in Fig.”
MethodHybrid Task Cascade for Instance Segmentation
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00511 · Cited by 1,587 · cited 2×
“We test eight state-of-the-art detectors listed as mask R-CNN (MASK) [138], Yolo v3 (YOLO) [139], cascade mask R-CNN (CAS) [140], Hybrid Task Cascade (HAC) [26], SDP [69], DPM [67], GT, and FRCNN [68] on MOT2017 train dataset to obtain different detection results, as shown in Fig.”
BackgroundMethodInfluentialTracking Without Bells and Whistles
Bergmann, P., Meinhardt, T., & Leal-Taixe, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00103 · Cited by 1,110
Globally-optimal greedy algorithms for tracking a variable number of objects
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995604 · Cited by 790
High-Speed tracking-by-detection without using image information
Bochinski, E., Eiselein, V., & Sikora, T. (2017) · 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2017.8078516 · Cited by 707 · cited 2×
“Moreover, we take these detection results as the input to data association and select four tracking algorithms (IOU [106], MOTDT [56], SST [141], and SORT [30]) for the testing purpose.”
BackgroundMethodMultiple Hypothesis Tracking Revisited
Kim, C., Li, F., Ciptadi, A., & Rehg, J. M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.533 · Cited by 666 · cited 2×
“In 2015, Kim et al. [54] utilized CNN to extract the 4096-dimensional feature for each detection box.”
MethodDeep learning in video multi-object tracking: A survey
Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., & Herrera, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.023 · Cited by 665 · cited 3×
“In the following, we discuss the main contributions and the limitations of the closely related surveys [1]–[6] on the MPT. • Luo et al. [1] presented the first comprehensive review about MOT.”
BackgroundMOTS: Multi-Object Tracking and Segmentation
Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B. B. G., Geiger, A., & Leibe, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00813 · Cited by 658
Yang, F., Choi, W., & Lin, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.234 · Cited by 608 · cited 9×
“Now, the MOT2017 includes the same videos as in MOT16, but contains three sets of detection for each video such as FRCNN [68], DPM [67] and SDP [69].”
MethodInfluentialChen, L., Ai, H., Zhuang, Z., & Shang, C. (2018) · 2018 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2018.8486597 · Cited by 530
Chu, Q., Ouyang, W., Li, H., Wang, X., Liu, B., & Yu, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.518 · Cited by 406
Robust Multi-Modality Multi-Object Tracking
Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00245 · Cited by 250
Chu, P., & Ling, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00627 · Cited by 246
30Hz Object Detection with DPM V5
Sadeghi, M. A., & Forsyth, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_5 · Cited by 107 · cited 5×
“Now, the MOT2017 includes the same videos as in MOT16, but contains three sets of detection for each video such as FRCNN [68], DPM [67] and SDP [69].”
MethodInfluential
Research works printed but not linked: 124
- 10.1109/CVPR.2012.6247893 · DOI · Paper page
- 10.1109/CVPR.2011.5995311 · DOI · Paper page
- 10.1109/CVPR.2008.4587577 · DOI
- shen, 2018, Tracklet association tracker: An end-to-end learning-based association approach for multi-object tracking, arXiv 1808 01562
- 10.1109/CVPR.2017.403 · DOI · Paper page
- 10.1109/TCSVT.2018.2825679 · DOI · Paper page
- 10.1109/CVPR.2013.472 · DOI · Paper page
- 10.1109/CVPR.2011.5995587 · DOI
- 10.1109/CVPR.2011.5995395 · DOI · Paper page
- 10.1109/CVPRW.2019.00105 · DOI · Paper page
- lin, 2014, Microsoft COCO: Common objects in context, Proc Eur Conf Comput Vis, 740
- 10.1007/978-3-642-33715-4_43 · DOI
- 10.1109/CVPR.2013.241 · DOI · Paper page
- 10.1109/CVPR.2015.7298718 · DOI
- 10.1109/CVPR.2015.7299193 · DOI
- 10.1007/978-3-642-33765-9_16 · DOI · Paper page
- 10.1109/ICCVW.2019.00025 · DOI
- 10.1109/CVPR.2017.292 · DOI · Paper page
- feng, 2019, Multi-object tracking with multiple cues and switcher-aware classification, arXiv 1901 06129
- 10.1109/TIP.2011.2159228 · doi:10.1109/tip.2011.2159228
- 10.1109/TCSVT.2020.2975842 · DOI
- 10.1109/IVS.2019.8813779 · DOI
- 10.1109/ICIP.2018.8451472 · DOI
- milan, 2017, Online multi-target tracking using recurrent neural networks, Proc AAAI Conf Artif Intell, 4225
- 10.1109/CVPR.2018.00232 · DOI
- 10.1109/CVPR.2015.7299178 · DOI · Paper page
- 10.1109/TPAMI.2015.2505309 · DOI · Paper page
- zhou, 2018, A survey of multi-object video tracking algorithms, Proc Adv Intell Sys Comput, 351, 10.2991/ijcis.2019.125905651 · doi:10.2991/ijcis.2019.125905651
- 10.1049/iet-cvi.2016.0178 · DOI
- 10.1049/iet-cvi.2018.5598 · DOI
- yoon, 2019, Online multiple pedestrian tracking using deep temporal appearance matching association, arXiv 1907 00831
- 10.1109/WMVC.2008.4544058 · DOI
- 10.1007/s13592-011-0060-6 · DOI
- 10.1109/CVPR.2017.394 · DOI · Paper page
- 10.1007/978-3-319-48881-3_8 · DOI · Paper page
- 10.1109/GlobalSIP.2016.7905816 · DOI
- xing, 2009, Multi-object tracking through occlusions by local tracklets filtering and global tracklets association with detection responses, Proc IEEE Conf Comput Vis Pattern Recognit, 1200
- zamir, 2012, GMCP-tracker: Global multi-object tracking using generalized minimum clique graphs, Proc Eur Conf Comput Vis, 343
- thoreau, 2018, Deep similarity metric learning for real-time pedestrian tracking, arXiv 1806 07592
- 10.1109/CVPR.2015.7299138 · DOI · Paper page
- 10.1109/CVPR.2015.7299036 · DOI · Paper page
- 10.1109/TPAMI.2008.57 · DOI · Paper page
- 10.1109/TPAMI.2016.2551245 · DOI
- 10.1016/j.patcog.2019.04.018 · DOI
- 10.1109/CVPRW.2018.00169 · DOI
- song, 2019, Online multi-object tracking framework with the GMPHD filter and occlusion group management, arXiv 1907 13347
- 10.1007/978-3-319-48881-3_2 · DOI · Paper page
- 10.1109/ICCVW.2019.00022 · DOI
- dendorfer, 2019, CVPR19 tracking and detection challenge: How crowded can it get?, arXiv 1906 04567
- 10.1007/978-3-319-48881-3_7 · DOI · Paper page
- 10.1109/LSP.2019.2940922 · DOI
- 10.4218/etrij.15.0114.0629 · DOI
- 10.1007/s11042-018-5781-3 · DOI
- 10.1109/CVPRW.2016.161 · DOI
- 10.1109/ACCESS.2018.2881019 · DOI
- tian, 2019, Online multi-object tracking using joint domain information in traffic scenarios, IEEE Trans Intell Transp Syst, 20, 1
- 10.1109/ICME.2018.8486454 · DOI · Paper page
- 10.1016/j.neucom.2015.07.028 · DOI
- 10.1109/CVPR.2017.206 · DOI · Paper page
- 10.1109/ICPCSI.2017.8392001 · DOI
- 10.1109/CVPRW.2018.00192 · DOI · Paper page
- 10.1109/TCSVT.2018.2882192 · DOI · Paper page
- 10.1109/ICCV.2017.322 · DOI · Paper page
- 10.1109/CVPRW.2017.266 · DOI · Paper page
- gautam, 2020, SDVTracker: Real-time multi-sensor association and tracking for self-driving vehicles, arXiv 2003 04447
- 10.1109/ICRAS49812.2020.9135065 · DOI
- 10.1109/ICSMC.2004.1400815 · DOI · Paper page
- yang, 2014, Aggregate channel features for multi-view face detection, Proc IEEE Int Joint Conf Biometrics, 1
- 10.1109/TPAMI.2019.2929520 · DOI · Paper page
- leal-taixé, 2015, MOTChallenge 2015: Towards a benchmark for multi-target tracking, arXiv 1504 01942 [cs]
- 10.1109/ACCESS.2019.2903121 · DOI
- hu, 2020, Multi-object tracking via end-to-end tracklet searching and ranking, arXiv 2003 02795
- 10.2528/PIERB15010503 · DOI
- 10.1109/LRA.2020.2974392 · DOI
- luo, 2014, Multiple object tracking: A literature review, arXiv 1409 7618 [cs]
- 10.1016/j.neucom.2018.02.068 · DOI
- 10.1109/TCSVT.2018.2856540 · DOI
- 10.1109/TPAMI.2017.2691769 · DOI · Paper page
- 10.1007/978-3-642-33783-3_8 · DOI
- wang, 2014, Coupled data association and L1 minimization for multiple object tracking under occlusion, Proc SPIE, 9273, 1
- 10.1109/ICIP.2014.7025168 · DOI
- 10.1145/3343031.3350853 · DOI · Paper page
- 10.1007/s11263-013-0666-4 · DOI
- 10.1145/3240508.3240548 · DOI
- 10.1109/WACV.2018.00128 · DOI
- 10.1109/ICCV.2019.00409 · DOI · Paper page
- 10.1109/CVPR.2016.461 · DOI · Paper page
- 10.1109/CVPR.2014.161 · DOI
- xiang, 2019, End-to-end learning deep CRF models for multi-object tracking, arXiv 1907 12176
- 10.1109/CVPRW.2018.00195 · DOI
- 10.1109/ROBIO.2015.7419004 · DOI
- 10.1007/978-81-322-1665-0_52 · DOI
- 10.1109/ICCV.2009.5459301 · DOI · Paper page
- 10.1109/CVPR.2010.5540102 · DOI · Paper page
- 10.1109/TPAMI.2017.2691768 · DOI · Paper page
- 10.1145/3343031.3350984 · DOI
- 10.1109/TMM.2018.2872897 · DOI
- 10.1109/TMM.2016.2605058 · DOI · Paper page
- 10.1007/978-3-642-33718-5_35 · DOI
- 10.1109/TPAMI.2013.221 · DOI
- 10.1109/CVPR.2017.142 · DOI · Paper page
- 10.1016/j.cviu.2016.12.003 · DOI
- 10.1016/j.cviu.2016.05.003 · DOI · Paper page
- xu, 2019, How to train your deep multi-object tracker, arXiv 1906 06618
- 10.1109/CVPR.2013.240 · DOI · Paper page
- 10.1109/TIP.2017.2745103 · DOI
- hu, 2012, Single and multiple object tracking using log-Euclidean Riemannian subspace and block-division appearance model, IEEE Trans Pattern Anal Mach Intell, 34, 2420, 10.1109/TPAMI.2012.42 · DOI · Paper page
- keuper, 2016, A multi-cut formulation for joint segmentation and tracking of multiple objects, arXiv 1607 06317
- 10.1109/ICCV.2017.278 · DOI
- 10.1109/CVPR.2008.4587584 · DOI · Paper page
- 10.1109/TPAMI.2018.2876253 · DOI · Paper page
- ma, 2018, Customized multi-person tracker, Vision Computer, 612
- 10.1109/TCSVT.2018.2881123 · DOI · Paper page
- 10.1109/CVPR.2017.495 · DOI · Paper page
- 10.1109/TPAMI.2013.103 · DOI · Paper page
- 10.1109/TIP.2018.2843129 · DOI
- 10.1109/ICCV.2015.534 · DOI · Paper page
- 10.1109/WACV.2019.00023 · DOI · Paper page
- 10.1109/ICIP.2017.8296360 · DOI · Paper page
- 10.1109/ICCV.2017.41 · DOI · Paper page
- 10.1007/978-3-030-01237-3_13 · DOI · Paper page
- 10.1109/WACV.2018.00057 · DOI · Paper page
- 10.1007/978-3-030-01219-9_36 · DOI
- 10.1109/ICCV.2009.5459278 · DOI · Paper page
Cited by
Parking Time Violation Tracking Using YOLOv8 and Tracking Algorithms
Sharma, N., Baral, S., Paing, M. P., & Chawuthai, R. (2023) · Sensors · doi:10.3390/s23135843 · Cited by 78
Metadata
- Authors
- Sun, Z., Chen, J., Chao, L., Ruan, W., & Mukherjee, M.
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2021-05
- Volume
- 31
- Issue
- 5
- Pages
- 1819-1833
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W3043157863
- Cited by
- 174
- References
- 22
- FWCI
- 6.43
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Funders
- National Nature Science Foundation of China (U1611461, 61876135, 61801335, 61672390, U1736206, U1903214)
- National Key Research and Development Program of China (2017YFC0803700)
- Hubei Technological Innovation Special Fund (2018AAA062, 2018CFA024, 2017AAA123, 2019CFB472)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.