No Blind Spots: Full-Surround Multi-Object Tracking for Autonomous Vehicles Using Cameras and LiDARs
Abstract
Online multi-object tracking (MOT) is extremely important for high-level spatial reasoning and path planning for autonomous and highly-automated vehicles. In this paper, we present a modular framework for tracking multiple objects (vehicles), capable of accepting object proposals from different sensor modalities (vision and range) and a variable number of sensors, to produce continuous object tracks. This work is a generalization of the MDP framework for MOT proposed by Xiang et al., with some key extensions - First, we track objects across multiple cameras and across different sensor modalities. This is done by fusing object proposals across sensors accurately and efficiently. Second, the objects of interest (targets) are tracked directly in the real world. This is a departure from traditional techniques where objects are simply tracked in the image plane. Doing so allows the tracks to be readily used by an autonomous agent for navigation and related tasks. To verify the effectiveness of our approach, we test it on real world highway data collected from a heavily sensorized testbed capable of capturing full-surround information. We demonstrate that our framework is well-suited to track objects through entire maneuvers around the ego-vehicle, some of which take more than a few minutes to complete. We also leverage the modularity of our approach by comparing the effects of including/excluding different sensors, changing the total number of sensors, and the quality of object proposals on the final tracking result.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The visual tracking task has attracted interest because of its relevance in applications including video surveillance and autonomous driving.
“The task has attracted a lot of interest because of its enormous relevance in many real-world applications, including video surveillance [82], autonomous driving [83], etc.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [83]
Cites
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
Algorithms for the Assignment and Transportation Problems
Munkres, J. R. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 4,001 · see citing works
Kalal, Z., Mikolajczyk, K., & Matas, J. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.239 · Cited by 3,357 · see citing works
Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics
Bernardin, K., & Stiefelhagen, R. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/246309 · Cited by 2,918 · see citing works
Robust object tracking with online multiple instance learning
Babenko, B., Yang, M.-H., & Belongie, S. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.226 · Cited by 2,007 · see citing works
A Boosted Particle Filter: Multitarget Detection and Tracking
Okuma, K., Taleghani, A., de Freitas, N., Little, J. J., & Lowe, D. G. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24670-1_3 · Cited by 1,022 · see citing works
Multiple Object Tracking Using K-Shortest Paths Optimization
Berclaz, J., Fleuret, F., Turetken, E., & Fua, P. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.21 · Cited by 1,015 · see citing works
Struck: Structured Output Tracking with Kernels
Hare, S., Golodetz, S., Saffari, A., Vineet, V., Cheng, M., Hicks, S. L., & Torr, P. H. S. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2509974 · Cited by 994 · see citing works
Global data association for multi-object tracking using network flows
Zhang, L., Li, Y., & Nevatia, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587584 · Cited by 945 · see citing works
Sivaraman, S., & Trivedi, M. M. (2013) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2013.2266661 · Cited by 911 · see citing works
Self-citationReal time robust L1 tracker using accelerated proximal gradient approach
Bao, C., Wu, Y., Ling, H., & Ji, H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247881 · Cited by 839 · see citing works
Globally-optimal greedy algorithms for tracking a variable number of objects
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995604 · Cited by 790 · see citing works
MCMC-based particle filtering for tracking a variable number of interacting targets
Khan, Z., Balch, T. R., & Dellaert, F. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.223 · Cited by 776 · see citing works
Learning to Track: Online Multi-object Tracking by Decision Making
Yu, X., Alahi, A., & Savarese, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.534 · Cited by 684 · see citing works
Learning to associate: HybridBoosted multi-target tracker for crowded scene
Li, Y., Huang, C., & Nevatia, R. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206735 · Cited by 633 · see citing works
Online Multiperson Tracking-by-Detection from a Single, Uncalibrated Camera
Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Gool, L. V. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.232 · Cited by 579 · see citing works
Continuous Energy Minimization for Multitarget Tracking
Milan, A., Roth, S., & Schindler, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.103 · Cited by 556 · see citing works
A multi-sensor fusion system for moving object detection and tracking in urban driving environments
Cho, H., Seo, Y., Kumar, B. V. K. V., & Rajkumar, R. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907100 · Cited by 441 · see citing works
Bae, S., & Yoon, K. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.159 · Cited by 415 · see citing works
Markov Chain Monte Carlo Data Association for Multi-Target Tracking
Oh, S., Russell, S., & Sastry, S. (2009) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2009.2012975 · Cited by 347 · see citing works
Multi-target tracking by on-line learned discriminative appearance models
Kuo, C.-H., Huang, C., & Nevatia, R. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540148 · Cited by 279 · see citing works
Multi-target Tracking by Lagrangian Relaxation to Min-cost Network Flow
Butt, A. A., & Collins, R. T. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.241 · Cited by 265 · see citing works
Asvadi, A., Premebida, C., Peixoto, P., & Nunes, U. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.06.007 · Cited by 243 · see citing works
Efficient representation of traffic scenes by means of dynamic stixels
Pfeiffer, D., & Franke, U. (2010) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2010.5548114 · Cited by 131 · see citing works
Multi-Target Tracking using a 3D-Lidar sensor for autonomous vehicles
Choi, J., Ulbrich, S., Lichte, B., et al. (2013) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2013.6728343 · Cited by 112 · see citing works
Challenges of Ground Truth Evaluation of Multi-target Tracking
Milan, A., Schindler, K., & Roth, S. (2013) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2013.111 · Cited by 96 · see citing works
Dynamic Probabilistic Drivability Maps for Lane Change and Merge Driver Assistance
Sivaraman, S., & Trivedi, M. M. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2309055 · Cited by 84 · see citing works
Self-citationDetection and Tracking of Moving Objects Using 2.5D Motion Grids
Asvadi, A., Peixoto, P., & Nunes, U. (2015) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2015.133 · Cited by 80 · see citing works
Vision-Based Multiple Interacting Targets Tracking via On-Line Supervised Learning
Song, X., Cui, J., Zha, H., & Zhao, H. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88690-7_48 · Cited by 77 · see citing works
3D object tracking using RGB and LIDAR data
Asvadi, A., Girão, P., Peixoto, P., & Nunes, U. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795718 · Cited by 77 · see citing works
Online Multi-target Tracking by Large Margin Structured Learning
Kim, S.-A., Kwak, S., Feyereisl, J., & Han, B. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37431-9_8 · Cited by 72 · see citing works
A full-3D voxel-based dynamic obstacle detection for urban scenario using stereo vision
Broggi, A., Cattani, S., Patander, M., Sabbatelli, M., & Zani, P. (2013) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2013.6728213 · Cited by 62 · see citing works
RefineNet: Refining Object Detectors for Autonomous Driving
Rajaram, R. N., Ohn-Bar, E., & Trivedi, M. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2017.2695896 · Cited by 51 · see citing works
Self-citationVatavu, A., Danescu, R. G., & Nedevschi, S. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2366248 · Cited by 48 · see citing works
Trajectories and Maneuvers of Surrounding Vehicles With Panoramic Camera Arrays
Dueholm, J., Kristoffersen, M. S., Satzoda, R., Moeslund, T., & Trivedi, M. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2016.2622921 · Cited by 42 · see citing works
Self-citationMulti-scale object candidates for generic object tracking in street scenes
Ošep, A., Hermans, A., Engelmann, F., Klostermann, D., Mathias, M., & Leibe, B. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487487 · Cited by 39 · see citing works
Efficient extraction of human motion volumes by tracking
Niebles, J. C., Han, B., & Fei-Fei, L. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540152 · Cited by 33 · see citing works
Fast and Robust Object Detection Using Visual Subcategories
Ohn-Bar, E., & Trivedi, M. M. (2014) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2014.32 · Cited by 31 · see citing works
Self-citationAllodi, M., Broggi, A., Giaquinto, D., et al. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535456 · Cited by 30 · see citing works
Object tracking with 3D LIDAR via multi-task sparse learning
Song, S., Xiang, Z., & Liu, J. (2015) · IEEE International Conference on Mechatronics and Automation (ICMA) · doi:10.1109/icma.2015.7237897 · Cited by 19 · see citing works
Rangesh, A., Yuen, K., Satzoda, R. K., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.07502 · Cited by 18 · see citing works
Self-citationGround Plane Polling for 6DoF Pose Estimation of Objects on the Road
Rangesh, A., & Trivedi, M. M. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.2966074 · Cited by 18 · see citing works
Self-citationVision-Based Front and Rear Surround Understanding Using Embedded Processors
Satzoda, R. K., Lee, S., Lu, F., et al. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2017.2686084 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Robust Multi-Modality Multi-Object Tracking
Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00245 · Cited by 250 · see citing works
Modeling Quality of IoT Experience in Autonomous Vehicles
Minovski, D., Ahlund, C., & Mitra, K. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2975418 · Cited by 59 · see citing works
Arikumar, K. S., Deepak Kumar, A., Gadekallu, T. R., Prathiba, S. B., & Tamilarasi, K. (2022) · Electronics · doi:10.3390/electronics11244203 · Cited by 46 · see citing works
- Authors
- Rangesh, A., & Trivedi, M. M.
- Affiliations
- University of California San Diego (US)
- Venue
- IEEE Transactions on Intelligent Vehicles
- Published
- 2019-12
- Volume
- 4
- Issue
- 4
- Pages
- 588-599
- ISSN
- 2379-8858 (ISSN-L)
- 2379-8904
- OpenAlex
- W2970938561
- Cited by
- 158
- References
- 44
- Citations by year
- 2026: 6 · 2025: 12 · 2024: 17 · 2023: 30 · 2022: 17 · 2021: 37 · 2020: 26 · 2019: 8 · 2018: 5
- FWCI
- 10.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Intelligent Vehicles (publishedVersion)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Target Tracking and Data Fusion in Sensor Networks
- Keywords
- multi-target tracking, autonomous vehicles, sensor fusion, object proposals, highway driving, vehicle tracking, sensor redundancy, real-world driving data
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review