Multiple Object Tracking with Correlation Learning
Abstract
Recent works have shown that convolutional networks have substantially improved the performance of multiple object tracking by simultaneously learning detection and appearance features. However, due to the local perception of the convolutional network structure itself, the long-range dependencies in both the spatial and temporal cannot be obtained efficiently. To incorporate the spatial layout, we propose to exploit the local correlation module to model the topological relationship between targets and their surrounding environment, which can enhance the discriminative power of our model in crowded scenes. Specifically, we establish dense correspondences of each spatial location and its context, and explicitly constrain the correlation volumes through self-supervised learning. To exploit the temporal context, existing approaches generally utilize two or more adjacent frames to construct an enhanced feature representation, but the dynamic motion scene is inherently difficult to depict via CNNs. Instead, our paper proposes a learnable correlation operator to establish frame-to-frame matches over convolutional feature maps in the different layers to align and propagate temporal context. With extensive experimental results on the MOT datasets, our approach demonstrates the effectiveness of correlation learning with the superior performance and obtains state-of-the-art MOTA of 76.5% and IDF1 of 73.6% on MOT17.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Only methods [50,53,54] that follow [66] and pretrain on 6 additional tracking datasets (6M) surpass TrackFormer's performance on MOT17 private detections.
“Only [50,53,54] which follow [66] and pretrain on 6 additional tracking datasets (6M) surpass our performance.”
From TrackFormer: Multi-Object Tracking with Transformers (2022) · cites this paper as [53]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · see citing works
Fully-Convolutional Siamese Networks for Object Tracking
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-48881-3_56 · Cited by 4,546 · see citing works
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · see citing works
Deformable ConvNets V2: More Deformable, Better Results
Zhu, X., Hu, H., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00953 · Cited by 2,943 · see citing works
MOT16: A Benchmark for Multi-Object Tracking
Milan, A., Leal-Taixe, L., Reid, I., Roth, S., & Schindler, K. (2016) · arXiv · doi:10.48550/arxiv.1603.00831 · Cited by 1,309 · see citing works
Towards Real-Time Multi-Object Tracking
Wang, Z., Zheng, L., Liu, Y., Li, Y., & Wang, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_7 · Cited by 1,195 · see citing works
MOTS: Multi-Object Tracking and Segmentation
Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B. B. G., Geiger, A., & Leibe, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00813 · Cited by 658 · see citing works
Detect to Track and Track to Detect
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.330 · Cited by 595 · see citing works
CrowdHuman: A Benchmark for Detecting Human in a Crowd
Shao, S., Zhao, Z., Li, B., Xiao, T., Yu, G., Zhang, X., & Sun, J. (2018) · arXiv · doi:10.48550/arxiv.1805.00123 · Cited by 534 · cited 1× · see citing works
“The backbone network is initialized with the parameters pre-trained on COCO [26] and then pre-trained on CrowdHuman [39] with self-supervised learning as FairMOT [58].”
MethodChen, L., Ai, H., Zhuang, Z., & Shang, C. (2018) · 2018 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2018.8486597 · Cited by 530 · see citing works
TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Model
Pang, B., Li, Y., Zhang, Y., Li, M., & Lu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00634 · Cited by 303 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
StrongSORT: Make DeepSORT Great Again
Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., & Meng, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3240881 · Cited by 1,191 · see citing works
“Recently, several joint tracking methods [57, 59, 28, 51] have been proposed to jointly train detection and other components, such as motion, embedding and association models.”
MethodTrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · see citing works
“9 1158 495 27813 108690 1572 TLR [52] CH+6M – 76.”
Method
- Authors
- Wang, Q., Zheng, Y., Pan, P., & Xu, Y.
- Affiliations
- Alibaba Group (Cayman Islands) (KY)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 3875-3885
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3176403636
- Cited by
- 215
- References
- 15
- Citations by year
- 2026: 12 · 2025: 35 · 2024: 60 · 2023: 56 · 2022: 44 · 2021: 7 · 2020: 1
- FWCI
- 13.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- multiple object tracking, correlation learning, spatial context modeling, temporal context modeling, self-supervised learning, dense correspondence, crowded scenes, MOT17
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
