Autonomous Vehicle Safety: An Interdisciplinary Challenge
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Koopman and Wagner showed that safety of AVs is a complex process that requires coordination between different disciplines starting from machine learning, to sensor development, to testing AVs, and developing a standard testing process [23].”
From Public acceptance and perception of autonomous vehicles: a comprehensive review (2021) · cites this paper as [23]
Citations
Cites
A few useful things to know about machine learning
Domingos, P. (2012) · Communications of the ACM · doi:10.1145/2347736.2347755 · Cited by 3,326 · cited 1×
“Another significant challenge is that machine learning techniques (Domingos 2012) such as classifiers that are widely used in fully autonomous vehicles (e.g., Aeberhard 2015) tend to be based on inductive training approaches rather than a more traditional requirements-based design process.”
BackgroundDeep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
An Inquiry concerning human understanding
Hume, D. (1955) · Cited by 1,775
Autonomous driving in urban environments: Boss and the Urban Challenge
Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M. N., Dolan, J., Duggins, D., Galatali, T., Geyer, C., et al. (2008) · Journal of Field Robotics · doi:10.1002/rob.20255 · Cited by 1,555
Towards fully autonomous driving: Systems and algorithms
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J. Z., Langer, D., Pink, O., Pratt, V., et al. (2011) · 2011 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2011.5940562 · Cited by 1,374 · cited 1×
“(e.g., Urmson 2008, Levinson 2011, Broggi 2013, Ziegler 2014, Aeberhard 2015, SAE J3018) However, it is well known that a testing-only approach is insufficient to ensure the safety of even non-autonomous software-based critical systems.”
BackgroundKalra, N., & Paddock, S. M. (2016) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2016.09.010 · Cited by 1,221 · cited 1×
“(Kane 2015) Such monitoring can be used during validation, on-road testing, and perhaps deployment to ensure that the vehicle does not exhibit unsafe behaviors, even if there are gaps or glitches in machine learning systems.”
BackgroundAn Enquiry Concerning Human Understanding
Hume, D. (1777) · David Hume: Enquiries Concerning Human Understanding and Concerning the Principles of Morals (Third Edition) · doi:10.1093/oseo/instance.00046350 · Cited by 972
Three Decades of Driver Assistance Systems: Review and Future Perspectives
Bengler, K., Dietmayer, K., Farber, B., Maurer, M., Stiller, C., & Winner, H. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2336271 · Cited by 868
Making Bertha Drive—An Autonomous Journey on a Historic Route
Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T., Franke, U., Appenrodt, N., Keller, C. G., et al. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2306552 · Cited by 854 · cited 1×
“(e.g., Urmson 2008, Levinson 2011, Broggi 2013, Ziegler 2014, Aeberhard 2015, SAE J3018) However, it is well known that a testing-only approach is insufficient to ensure the safety of even non-autonomous software-based critical systems.”
BackgroundSoftware Engineering for Self-Adaptive Systems: A Second Research Roadmap
de Lemos, R., Giese, H., Müller, H. A., Shaw, M., Andersson, J., Litoiu, M., Schmerl, B., Tamura, G., Villegas, N. M., Vogel, T., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35813-5_1 · Cited by 716 · cited 2×
“…planning, finding a way to validate inductive-based learning strategies, providing resilience in the face of likely gaps in early-deployed system requirements, and having an appropriate safety certification strategy to demonstrate that a sufficient level of safety has actually been achieved.”
BackgroundChallenges in autonomous vehicle testing and validation
Koopman, P., & Wagner, M. (2016) · SAE International Journal of Transportation Safety · doi:10.4271/2016-01-0128 · Cited by 675 · cited 5×
“But it also means things such as fail-over mission planning, finding a way to validate inductive-based learning strategies, providing resilience in the face of likely gaps in early-deployed system requirements, and having an appropriate safety certification strategy to demonstrate that a sufficient…”
BackgroundInfluentialSelf-citationAutonomous Vehicle Technology: A Guide for Policymakers
(2014) · RAND Corporation eBooks · doi:10.7249/rr443 · Cited by 589
Software safety: why, what, and how
Leveson, N. G. (1986) · ACM Computing Surveys · doi:10.1145/7474.7528 · Cited by 471 · cited 1×
“(Leveson 1986) Current software safety approaches such as the ISO 26262 “V” process typically assume that high-quality requirements are refined into an implementation.”
BackgroundThe infeasibility of quantifying the reliability of life-critical real-time software
Butler, R. W., & Finelli, G. B. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.210303 · Cited by 398
Reliability Issues in Computing System Design
Randell, B., Lee, P., & Treleaven, P. C. (1978) · ACM Computing Surveys · doi:10.1145/356725.356729 · Cited by 383
Autonomous Ground Vehicles—Concepts and a Path to the Future
Luettel, T., Himmelsbach, M., & Wuensche, H.-J. (2012) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2189803 · Cited by 337 · cited 1×
“…approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory…”
BackgroundKalra, N., & Paddock, S. (2016) · RAND Corporation eBooks · doi:10.7249/rr1478 · Cited by 305
Experience, Results and Lessons Learned from Automated Driving on Germany's Highways
Aeberhard, M., Rauch, S., Bahram, M., Tanzmeister, G., Thomas, J., Pilat, Y., Homm, F., Huber, W., & Kaempchen, N. (2015) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2360306 · Cited by 250 · cited 2×
“(e.g., Urmson 2008, Levinson 2011, Broggi 2013, Ziegler 2014, Aeberhard 2015, SAE J3018) However, it is well known that a testing-only approach is insufficient to ensure the safety of even non-autonomous software-based critical systems.”
BackgroundA Comparative Study of Probabilistic Roadmap Planners
Geraerts, R., & Overmars, M. H. (2004) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-540-45058-0_4 · Cited by 240 · cited 1×
“What really matters is that the machine learning validation set has to be comprehensive enough to make sure that there are no gaps in system behavior.”
BackgroundExtensive Tests of Autonomous Driving Technologies
Broggi, A., Buzzoni, M., Debattisti, S., Grisleri, P., Laghi, M. C., Medici, P., & Versari, P. (2013) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2013.2262331 · Cited by 205 · cited 1×
“(e.g., Urmson 2008, Levinson 2011, Broggi 2013, Ziegler 2014, Aeberhard 2015, SAE J3018) However, it is well known that a testing-only approach is insufficient to ensure the safety of even non-autonomous software-based critical systems.”
BackgroundLearning Driver Behavior Models from Traffic Observations for Decision Making and Planning
Gindele, T., Brechtel, S., & Dillmann, R. (2015) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2357038 · Cited by 199 · cited 1×
“(Butler 1993) More than just testing is required for full scale deployment of any safety-critical automotive software.”
Background(2012) · Santa Clara Law Digital Commons (Santa Clara University) · Cited by 139
The infeasibility of experimental quantification of life-critical software reliability
Butler, R. W., & Finelli, G. B. (1991) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/123041.123054 · Cited by 133
Cooperative (rather than autonomous) vehicle-highway automation systems
Shladover, S. E. (2009) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2009.932716 · Cited by 126 · cited 2×
“Even if there are dedicated lanes initially to ease deployment (Shladover 2009), over time it seems likely that there will be tremendous public pressure to spread autonomy to mixed autonomous/human driven vehicle scenarios.”
BackgroundInverting Convolutional Networks with Convolutional Networks.Preprint
Dosovitskiy, A., & Brox, T. (2015) · arXiv (Cornell University) · Cited by 75
A Case Study on Runtime Monitoring of an Autonomous Research Vehicle (ARV) System
Kane, A., Chowdhury, O., Datta, A., & Koopman, P. J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23820-3_7 · Cited by 63
Self-citationAutonomous Driving - A Practical Roadmap
Rupp, J. D., & King, A. G. (2010) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2010-01-2335 · Cited by 58
Active learning from demonstration for robust autonomous navigation
Silver, D., Bagnell, J. A., & Stentz, A. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6224757 · Cited by 52
Safety Verification of Automated Driving Systems
Kianfar, R., Falcone, P., & Fredriksson, J. (2013) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2013.2278405 · Cited by 48 · cited 1×
“As everyone gains more experience with the technology, no doubt a few more high-level problems and many more detailed issues will emerge, but this is a starting point for understanding the bigger picture.”
BackgroundA Comparative Study of Probabilistic Roadmap Planners.
Gerærts, R., & Overmars, M. H. (2002) · Cited by 46
Design by extrapolation: an evaluation of fault-tolerant avionics
Hammett, R. (2002) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc.2001.963314 · Cited by 31 · cited 2×
“IEEE IntEllIgEnt transportatIon systEms magazInE • 93 • spring 2017IEEE IntEllIgEnt transportatIon systEms magazInE • 92 • spring 2017 they will be exposed to real-world data that differs somehow from training and validation data.”
BackgroundTest oracle automation for V&V of an autonomous spacecraft's planner
Feather, M. S., & Smith, B. (2001) · Cited by 4
Research works printed but not linked: 11
- 2015, Guidelines for safe on-road testing of SAE Level 3 4 and 5 prototype Automated Driving Systems (ADS)
- parent, 2013, Legal issues and certification of the fully automated vehicles: Best practices and lessons learned, CityMobil2 Rep
- 2014, Autonomous driving pilot study, Tech Rep TSG 2014-1316
- 2016, Surface vehicle recommended practice Cybersecurity guidebook for cyber-physical vehicle systems
- 1988, System Design and Analysis FAA
- gomes, 2014, Hidden obstacles for Google's self-driving cars, MIT Technol Rev
- 1995, Malfunction management activity area report for AHS health management, AHS Precursor Task E report
- 2011, Road Vehicles-Functional Safety
- 2015, Self-driving vehicles in an urban context press briefing
- 2013, Preliminary Statement of Policy Concerning Automated Vehicles
- learner, 2015, The hurdles facing autonomous vehicles, Automobile
Cited by
A Survey on the Internet of Things (IoT) Forensics: Challenges, Approaches, and Open Issues
Stoyanova, M., Nikoloudakis, Y., Panagiotakis, S., Pallis, E., & Markakis, E. K. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2962586 · Cited by 852
“After all, the social acceptance of autonomous vehicles lies on the premise that autonomous vehicles will, on the whole, be safer drivers than people [149].”
BackgroundAutonomous Cars: Research Results, Issues, and Future Challenges
Hussain, R., & Zeadally, S. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2018.2869360 · Cited by 631
“[146] and Koopman and Wagner [147] argued for novel methods for safety validation for the autonomous car system.”
BackgroundMethodInfluentialTrustworthy artificial intelligence
Thiebes, S., Lins, S., & Sunyaev, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00441-4 · Cited by 595
“…non-maleficence principle especially investigated means for the development and deployment of safe and secure AI in the areas of autonomous driving (Koopman and Wagner 2017) and medicine (Wiens et al. 2019), the non-maleficence principle is highly important for electronic markets due to the…”
BackgroundSurvey on Scenario-Based Safety Assessment of Automated Vehicles
Riedmaier, S., Ponn, T., Ludwig, D., Schick, B., & Diermeyer, F. (2020) · IEEE Access · doi:10.1109/access.2020.2993730 · Cited by 481
Public acceptance and perception of autonomous vehicles: a comprehensive review
Othman, K. (2021) · AI and Ethics · doi:10.1007/s43681-021-00041-8 · Cited by 347
Metadata
- Authors
- Koopman, P., & Wagner, M.
- Affiliations
- Carnegie Mellon University (US)
- Edge Case Research (United States) (US)
- ORCID
- Koopman, Philip
- Venue
- IEEE Intelligent Transportation Systems Magazine
- Published
- 2017
- Volume
- 9
- Issue
- 1
- Pages
- 90-96
- ISSN
- 1939-1390 (ISSN-L)
- OpenAlex
- W2576845151
- Cited by
- 688
- References
- 34
- Citations by year
- 2026: 61 · 2025: 91 · 2024: 88 · 2023: 86 · 2022: 80 · 2021: 95 · 2020: 90 · 2019: 49 · 2018: 38 · 2017: 8
- FWCI
- 73.65
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Intelligent Transportation Systems Magazine (publishedVersion)
- Topics
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- autonomous vehicles, vehicle safety, hardware fault tolerance, human-vehicle interaction, regulatory framework, inductive learning, dependability, fleet deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.