The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
Abstract
Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This monograph attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Self-citationRobust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Self-citationObject Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
LSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450
LOAM: Lidar Odometry and Mapping in Real-time
Zhang, J., & Singh, S. (2014) · Robotics: Science and Systems X · doi:10.15607/rss.2014.x.007 · Cited by 3,303
A volumetric method for building complex models from range images
Curless, B., & Levoy, M. (1996) · Proceedings of the 23rd annual conference on Computer graphics and interactive techniques · doi:10.1145/237170.237269 · Cited by 3,124
PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
Kendall, A., Grimes, M., & Cipolla, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.336 · Cited by 2,489
A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.19 · Cited by 2,457
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632
A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection
Cai, Z., Fan, Q., Feris, R. S., & Vasconcelos, N. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_22 · Cited by 1,605
Scaramuzza, D., & Fraundorfer, F. (2011) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2011.943233 · Cited by 1,550
High-accuracy stereo depth maps using structured light
Scharstein, D., & Szeliski, R. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211354 · Cited by 1,542
FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance
Cummins, M., & Newman, P. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364908090961 · Cited by 1,472
iSAM2: Incremental smoothing and mapping using the Bayes tree
Kaess, M., Johannsson, H., Roberts, R., Ila, V., Leonard, J. J., & Dellaert, F. (2012) · The International Journal of Robotics Research · doi:10.1177/0278364911430419 · Cited by 1,362
MOT16: A Benchmark for Multi-Object Tracking
Milan, A., Leal-Taixe, L., Reid, I., Roth, S., & Schindler, K. (2016) · arXiv · doi:10.48550/arxiv.1603.00831 · Cited by 1,309
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.04412 · Cited by 1,273
Monocular Pedestrian Detection: Survey and Experiments
Enzweiler, M., & Gavrila, D. M. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.260 · Cited by 1,213
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174
Visual Place Recognition: A Survey
Lowry, S., Sünderhauf, N., Newman, P., Leonard, J. J., Cox, D., Corke, P., & Milford, M. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2496823 · Cited by 1,079
iSAM: Incremental Smoothing and Mapping
Kaess, M., Ranganathan, A., & Dellaert, F. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2006706 · Cited by 1,076
Monocular 3D Object Detection for Autonomous Driving
Chen, X., Kundu, K., Zhang, Z., Ma, H., Fidler, S., & Urtasun, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.236 · Cited by 1,019
A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
Schops, T., Schonberger, J. L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., & Geiger, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.272 · Cited by 915
Self-citationMaking Bertha Drive—An Autonomous Journey on a Historic Route
Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T., Franke, U., Appenrodt, N., Keller, C. G., et al. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2306552 · Cited by 854
Is Faster R-CNN Doing Well for Pedestrian Detection?
Zhang, L., Lin, L., Liang, X., & He, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_28 · Cited by 852
Globally-optimal greedy algorithms for tracking a variable number of objects
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995604 · Cited by 790
A mobile vision system for robust multi-person tracking
Ess, A., Leibe, B., Schindler, K., & Van Gool, L. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587581 · Cited by 721
Multiple Hypothesis Tracking Revisited
Kim, C., Li, F., Ciptadi, A., & Rehg, J. M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.533 · Cited by 666
Yang, F., Choi, W., & Lin, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.234 · Cited by 608
Vehicle Detection from 3D Lidar Using Fully Convolutional Network
Li, B., Zhang, T., & Xia, T. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.042 · Cited by 577
Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation
Ghiasi, G., & Fowlkes, C. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_32 · Cited by 502
Map-Based Precision Vehicle Localization in Urban Environments
Levinson, J., Montemerlo, M., & Thrun, S. (2007) · Robotics: Science and Systems III · doi:10.15607/rss.2007.iii.016 · Cited by 477
Dynamic monocular machine vision
Dickmanns, E. D., & Graefe, V. (1988) · Machine Vision and Applications · doi:10.1007/bf01212361 · Cited by 353
Multi-cue onboard pedestrian detection
Wojek, C., Walk, S., & Schiele, B. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206638 · Cited by 350
Voting for Voting in Online Point Cloud Object Detection
Zeng Wang, D., & Posner, I. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.035 · Cited by 337
Double window optimisation for constant time visual SLAM
Strasdat, H., Davison, A. J., Montiel, J. M. M., & Konolige, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126517 · Cited by 296
Appearance-Guided Monocular Omnidirectional Visual Odometry for Outdoor Ground Vehicles
Scaramuzza, D., & Siegwart, R. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2004490 · Cited by 292
Joint Semantic Segmentation and 3D Reconstruction from Monocular Video
Kundu, A., Li, Y., Dellaert, F., Li, F., & Rehg, J. M. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10599-4_45 · Cited by 220
Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction
Vineet, V., Miksik, O., Lidegaard, M., Niebner, M., Golodetz, S., Prisacariu, V. A., Kahler, O., Murray, D. W., Izadi, S., Peerez, P., et al. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7138983 · Cited by 200
TorontoCity: Seeing the World with a Million Eyes
Wang, S., Bai, M., Mattyus, G., Chu, H., Luo, W., Yang, B., Liang, J., Cheverie, J., Fidler, S., & Urtasun, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.327 · Cited by 186
Dense Reconstruction Using 3D Object Shape Priors
Dame, A., Prisacariu, V. A., Ren, C. Y., & Reid, I. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.170 · Cited by 135
Regionlets for Generic Object Detection
Wang, X., Yang, M., Zhu, S., & Lin, Y. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2389830 · Cited by 107
Blaha, M., Vogel, C., Richard, A., Wegner, J. D., Pock, T., & Schindler, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.346 · Cited by 88
Continuous-Time Trajectory Estimation for Event-based Vision Sensors
Mueggler, E., Gallego, G., & Scaramuzza, D. (2015) · Robotics: Science and Systems XI · doi:10.15607/rss.2015.xi.036 · Cited by 49
Integrating metric and semantic maps for vision-only automated parking
Grimmett, H., Buerki, M., Paz, L., Pinies, P., Furgale, P., Posner, I., & Newman, P. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139484 · Cited by 29
Cited by
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809
A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit
Padilla, R., Passos, W. L., Dias, T. L. B., Netto, S. L., & da Silva, E. A. B. (2021) · Electronics · doi:10.3390/electronics10030279 · Cited by 735
Autonomous driving system: A comprehensive survey
Zhao, J., Zhao, W., Deng, B., Wang, Z., Zhang, F., Zheng, W., Cao, W., Nan, J., Lian, Y., & Burke, A. F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122836 · Cited by 361
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
On the Robustness of Semantic Segmentation Models to Adversarial Attacks
Arnab, A., Miksik, O., & Torr, P. H. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00099 · Cited by 319
Metadata
- Authors
- Janai, J., Güney, F., Behl, A., & Geiger, A.
- Publisher
- Now Publishers
- Published
- 2020
- OpenAlex
- W2609532991
- Cited by
- 232
- References
- 61
- FWCI
- 16.66
- Type
- book
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.