Robust Multi-Modality Multi-Object Tracking
Open accessAbstract
Multi-sensor perception is crucial to ensure the reliability and accuracy in autonomous driving system, while multi-object tracking (MOT) improves that by tracing sequential movement of dynamic objects. Most current approaches for multi-sensor multi-object tracking are either lack of reliability by tightly relying on a single input source (e.g., center camera), or not accurate enough by fusing the results from multiple sensors in post processing without fully exploiting the inherent information. In this study, we design a generic sensor-agnostic multi-modality MOT framework (mmMOT), where each modality (i.e., sensors) is capable of performing its role independently to preserve reliability, and could further improving its accuracy through a novel multi-modality fusion module. Our mmMOT can be trained in an end-to-end manner, enables joint optimization for the base feature extractor of each modality and an adjacency estimator for cross modality. Our mmMOT also makes the first attempt to encode deep representation of point cloud in data association process in MOT. We conduct extensive experiments to evaluate the effectiveness of the proposed framework on the challenging KITTI benchmark and report state-of-the-art performance. Code and models are available at https://github.com/ZwwWayne/mmMOT.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multimodal multi-object tracking is a universal aspect of multi-object tracking aimed at ensuring the accuracy of autonomous systems by mapping the motion sequence of dynamic objects.
“Multimodal MOT is a universal aspect of MOT aimed at ensuring the accuracy of autonomous systems by mapping the motion sequence of dynamic objects [194].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [194]
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
(2017) · Cited by 11,625 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
PointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics
Bernardin, K., & Stiefelhagen, R. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/246309 · Cited by 2,918 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks
Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., & Yan, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00441 · Cited by 2,607 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Super-convergence: very fast training of neural networks using large learning rates
Smith, L. N., & Topin, N. (2019) · Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications · doi:10.1117/12.2520589 · Cited by 1,527 · see citing works
Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
Bell, S., Zitnick, C. L., Bala, K., & Girshick, R. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.314 · Cited by 1,344 · see citing works
Wu, B., Wan, A., Yue, X., & Keutzer, K. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462926 · Cited by 967 · see citing works
Wu, B., Zhou, X., Zhao, S., Yue, X., & Keutzer, K. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793495 · Cited by 761 · see citing works
Learning to Track: Online Multi-object Tracking by Decision Making
Yu, X., Alahi, A., & Savarese, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.534 · Cited by 684 · see citing works
Luo, W., Yang, B., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00376 · Cited by 644 · see citing works
Learning to associate: HybridBoosted multi-target tracker for crowded scene
Li, Y., Huang, C., & Nevatia, R. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206735 · Cited by 633 · see citing works
Online Multiperson Tracking-by-Detection from a Single, Uncalibrated Camera
Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Gool, L. V. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.232 · Cited by 579 · see citing works
Tracking the Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies
Sadeghian, A., Alahi, A., & Savarese, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.41 · Cited by 567 · see citing works
The Perfect Match: 3D Point Cloud Matching With Smoothed Densities
Gojčič, Ž., Zhou, C., Wegner, J. D., & Wieser, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00569 · Cited by 530 · see citing works
A multi-sensor fusion system for moving object detection and tracking in urban driving environments
Cho, H., Seo, Y., Kumar, B. V. K. V., & Rajkumar, R. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907100 · Cited by 441 · see citing works
GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs
Zamir, A., Dehghan, A., & Shah, M. A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_25 · Cited by 360 · see citing works
Multiple Sensor Fusion and Classification for Moving Object Detection and Tracking
Chavéz-García, R. O., & Aycard, O. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2479925 · Cited by 358 · see citing works
Luo, W., Yang, B., & Urtasun, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12395 · Cited by 345 · see citing works
PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic SegmentationPreprint
Jiang, M., Wu, Y., Zhao, T., Zhao, Z., & Lu, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00652 · Cited by 338 · see citing works
Accurate Single Stage Detector Using Recurrent Rolling Convolution
Ren, J., Chen, X., Liu, J., Sun, W., Pang, J., Yan, Q., Tai, Y.-W., & Xu, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.87 · Cited by 305 · see citing works
IntentNet: Learning to Predict Intention from Raw Sensor DataPreprint
Casas, S., Luo, W., & Urtasun, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.07907 · Cited by 248 · see citing works
Deep Network Flow for Multi-object Tracking
Schulter, S., Vernaza, P., Choi, W., & Chandraker, M. K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.292 · Cited by 213 · see citing works
Subgraph decomposition for multi-target tracking
Tang, S., Andres, B., Andriluka, M., & Schiele, B. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299138 · Cited by 195 · see citing works
Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking
Sharma, S., Ansari, J. A., Jatavallabhula, K. M., & Krishna, K. M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461018 · Cited by 169 · see citing works
No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles Using Cameras and LiDARs
Rangesh, A., & Trivedi, M. M. (2019) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2938110 · Cited by 158 · see citing works
Mono-Camera 3D Multi-Object Tracking Using Deep Learning Detections and PMBM Filtering
Scheidegger, S., Benjaminsson, J., Rosenberg, E., Krishnan, A., & Granström, K. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500454 · Cited by 142 · see citing works
End-to-end Learning of Multi-sensor 3D Tracking by Detection
Frossard, D., & Urtasun, R. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462884 · Cited by 135 · see citing works
Occlusion Geodesics for Online Multi-object Tracking
Possegger, H., Mauthner, T., Roth, P. M., & Bischof, H. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.170 · Cited by 114 · see citing works
Combined image- and world-space tracking in traffic scenes
Ošep, A., Mehner, W., Mathias, M., & Leibe, B. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989230 · Cited by 114 · see citing works
Multi-Target Tracking using a 3D-Lidar sensor for autonomous vehicles
Choi, J., Ulbrich, S., Lichte, B., et al. (2013) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2013.6728343 · Cited by 112 · see citing works
FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation
Lenz, P., Geiger, A., & Urtasun, R. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.496 · Cited by 105 · see citing works
Deep Multi-Sensor Lane Detection
Bai, M., Máttyus, G., Homayounfar, N., Wang, S., Lakshmikanth, S. K., & Urtasun, R. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594388 · Cited by 102 · see citing works
Online Multi-Object Tracking Using Joint Domain Information in Traffic Scenarios
Tian, W., Lauer, M., & Chen, L. F. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2892413 · Cited by 97 · see citing works
FishNet: A Versatile Backbone for Image, Region, and Pixel Level PredictionPreprint
Sun, S., Pang, J., Shi, J., Yi, S., & Ouyang, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.03495 · Cited by 86 · see citing works
Self-citationZhou, H., Ouyang, W., Cheng, J., Wang, X., & Li, H. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2825679 · Cited by 84 · see citing works
Self-citationDetection and Tracking of Moving Objects Using 2.5D Motion Grids
Asvadi, A., Peixoto, P., & Nunes, U. (2015) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2015.133 · Cited by 80 · see citing works
3D object tracking using RGB and LIDAR data
Asvadi, A., Girão, P., Peixoto, P., & Nunes, U. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795718 · Cited by 77 · see citing works
A Lightweight Online Multiple Object Vehicle Tracking Method
Gündüz, G., & Acarman, T. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500386 · Cited by 38 · see citing works
Object tracking with 3D LIDAR via multi-task sparse learning
Song, S., Xiang, Z., & Liu, J. (2015) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma.2015.7237897 · Cited by 19 · see citing works
No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles using Cameras & LiDARsPreprint
Rangesh, A., & Trivedi, M. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08755 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
- Authors
- Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C.
- Affiliations
- Nanyang Technological University (SG)
- Sensetime (China) (CN)
- University of Oxford (GB)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 2365-2374
- OpenAlex
- W2983227562
- Cited by
- 250
- References
- 53
- Citations by year
- 2026: 10 · 2025: 33 · 2024: 39 · 2023: 45 · 2022: 44 · 2021: 43 · 2020: 35 · 2019: 1
- FWCI
- 30.60
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministry of Education, India
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Target Tracking and Data Fusion in Sensor Networks
- Keywords
- multi-target tracking, multi-sensor fusion, autonomous driving perception, point cloud representation, data association, end-to-end learning, KITTI dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
