On Offline Evaluation of Vision-Based Driving Models
Open accessAbstract
Autonomous driving models should ideally be evaluated by deploying them on a fleet of physical vehicles in the real world. Unfortunately, this approach is not practical for the vast majority of researchers. An attractive alternative is to evaluate models offline, on a pre-collected validation dataset with ground truth annotation. In this paper, we investigate the relation between various online and offline metrics for evaluation of autonomous driving models. We find that offline prediction error is not necessarily correlated with driving quality, and two models with identical prediction error can differ dramatically in their driving performance. We show that the correlation of offline evaluation with driving quality can be significantly improved by selecting an appropriate validation dataset and suitable offline metrics.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles
Paden, B. A., Čáp, M., Yong, S. Z., Yershov, D. S., & Frazzoli, E. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2016.2578706 · Cited by 2,601 · see citing works
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338 · see citing works
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,928 · see citing works
Self-citationDeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · see citing works
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632 · see citing works
ALVINN, an autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · doi:10.1184/r1/6603146 · Cited by 1,403 · see citing works
VirtualWorlds as Proxy for Multi-object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.470 · Cited by 1,079 · see citing works
Traffic-Sign Detection and Classification in the Wild
Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., & Hu, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.232 · Cited by 1,062 · see citing works
Simultaneous Localization and Mapping: A Survey of Current Trends in Autonomous Driving
Bresson, G., Alsayed, Z., Yu, L., & Glaser, S. (2017) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2017.2749181 · Cited by 885 · see citing works
End-to-End Learning of Driving Models from Large-Scale Video Datasets
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.376 · Cited by 884 · see citing works
Neural Information Processing Systems (NIPS)
Shang, X., Zhu, Z., Leimkuhler, B., & Storkey, A. (2015) · Cited by 831 · see citing works
(2005) · neural information processing systems · Cited by 446 · see citing works
Richter, S. R., Hayder, Z., & Koltun, V. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.243 · Cited by 434 · see citing works
Self-citationVirtual Worlds as Proxy for Multi-Object Tracking AnalysisPreprint
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06457 · Cited by 302 · see citing works
Uncertainty-Aware Reinforcement Learning for Collision AvoidancePreprint
Kahn, G., Villaflor, A., Pong, V. H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01182 · Cited by 228 · see citing works
Learning a Driving SimulatorPreprint
Santana, E., & Hotz, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.01230 · Cited by 226 · see citing works
End-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129 · see citing works
Self-citationVision-Based Offline-Online Perception Paradigm for Autonomous Driving
Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vázquez, D., & López, A. M. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.38 · Cited by 123 · see citing works
Self-citationSemantic Stixels: Depth is not enough
Schneider, L., Cordts, M., Rehfeld, T., Pfeiffer, D., Enzweiler, M., Franke, U., Pollefeys, M., & Roth, S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535373 · Cited by 90 · see citing works
Predicting Scene Parsing and Motion Dynamics in the FuturePreprint
Jin, X., Xiao, H., Shen, X., Yan, S., Lin, Z., Chen, Y., Jie, Z., Feng, J., & Yan, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.03270 · Cited by 62 · see citing works
Going Deeper: Autonomous Steering with Neural Memory Networks
Fernando, T., Denman, S., Sridharan, S., et al. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.34 · Cited by 52 · see citing works
Adding navigation to the equation: Turning decisions for end-to-end vehicle control
Hubschneider, C., Bauer, A., Weber, M., & Zöllner, J. M. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317923 · Cited by 41 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Dosovitskiy, A., Ros, G., Codevilla, F., López, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning (CoRL) (2017)
- Ebrahimi, S., Rohrbach, A., Darrell, T.: Gradient-free policy architecture search and adaptation. In: Conference on Robot Learning (CoRL) (2017)
Cited by
Sorted by citations
Multimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319 · see citing works
Self-citationA Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
- Authors
- Codevilla, F., López, A. M., Koltun, V., & Dosovitskiy, A.
- Affiliations
- Universitat Autònoma de Barcelona (ES)
- Intel (United States) (US)
- Intel (Germany) (DE)
- Corresponding author
- Dosovitskiy, Alexey
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2018
- Pages
- 246-262
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2890235476
- Semantic Scholar
- 206c0d1c6cb90b3914a25de04c5bb17182f6696b
- Cited by
- 103
- References
- 26
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 16 · 2023: 11 · 2022: 12 · 2021: 27 · 2020: 15 · 2019: 7 · 2018: 1
- FWCI
- 12.82
- Type
- conference paper
- Language
- en
- Open access
- green
- ddd.uab.cat
- Licence
- public-domain
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, public-domain)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- autonomous driving, offline evaluation, prediction error, ground truth annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
