Requirements-driven test generation for autonomous vehicles with machine learning components
Open accessAbstract
Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically identify test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and system-level behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is difficult and mostly impossible to characterize all the behaviors of ML components under all circumstances.
“It is difficult and mostly impossible to characterize all the behaviors of ML components under all circumstances (Tuncali et al., 2019).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Tuncali et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408 · see citing works
The unscented Kalman filter for nonlinear estimation
Wan, E. A., & van der Merwe, R. (2002) · IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (AS-SPCC) · doi:10.1109/asspcc.2000.882463 · Cited by 4,373 · see citing works
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · see citing works
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495 · see citing works
Schubert, E., Sander, J., Ester, M., Kriegel, H., & Xu, X. (2017) · ACM Transactions on Database Systems · doi:10.1145/3068335 · Cited by 2,789 · see citing works
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · see citing works
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180220 · Cited by 1,400 · see citing works
Cyberbotics Ltd. Webots <sup>™</sup> : Professional Mobile Robot Simulation
Michel, O. (2004) · International Journal of Advanced Robotic Systems · doi:10.5772/5618 · Cited by 1,261 · see citing works
Challenges in autonomous vehicle testing and validation
Koopman, P., & Wagner, M. (2016) · SAE International Journal of Transportation Safety · doi:10.4271/2016-01-0128 · Cited by 675 · see citing works
What's Decidable about Hybrid Automata?
Henzinger, T. A., Kopke, P. W., Puri, A., et al. (1998) · Journal of Computer and System Sciences · doi:10.1006/jcss.1998.1581 · Cited by 669 · see citing works
Wu, B., Wan, A., Iandola, F., Jin, P. H., & Keutzer, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.60 · Cited by 614 · see citing works
Robustness of temporal logic specifications for continuous-time signals
Fainekos, G., & Pappas, G. J. (2009) · Theoretical Computer Science · doi:10.1016/j.tcs.2009.06.021 · Cited by 611 · see citing works
Self-citationOn a Formal Model of Safe and Scalable Self-driving CarsPreprint
Shalev‐Shwartz, S., Shammah, S., & Shashua, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.06374 · Cited by 540 · see citing works
Breach, A Toolbox for Verification and Parameter Synthesis of Hybrid Systems
Donzé, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-14295-6_17 · Cited by 460 · see citing works
Zhao, D. X., Lam, H., Peng, H., et al. (2016) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2016.2582208 · Cited by 457 · see citing works
S-TaLiRo: A Tool for Temporal Logic Falsification for Hybrid Systems
Annpureddy, Y., Liu, C., Fainekos, G., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19835-9_21 · Cited by 435 · see citing works
Self-citationModel based vehicle detection and tracking for autonomous urban driving
Petrovskaya, A., & Thrun, S. (2009) · Autonomous Robots · doi:10.1007/s10514-009-9115-1 · Cited by 430 · see citing works
A committee of neural networks for traffic sign classification
Ciresan, D. C., Meier, U., Masci, J., & Schmidhuber, J. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033458 · Cited by 379 · see citing works
Specification-Based Monitoring of Cyber-Physical Systems: A Survey on Theory, Tools and Applications
Bartocci, E., Deshmukh, J. V., Donzé, A., Fainekos, G., Maler, O., Ničković, D., & Sankaranarayanan, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75632-5_5 · Cited by 333 · see citing works
Self-citationIn the Passenger Seat: Investigating Ride Comfort Measures in Autonomous Cars
Elbanhawi, M., Simić, M., & Jazar, R. N. (2015) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2015.2405571 · Cited by 330 · see citing works
Feature-Guided Black-Box Safety Testing of Deep Neural Networks
Wicker, M., Huang, X., & Kwiatkowska, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-89960-2_22 · Cited by 223 · see citing works
Tuncali, C. E., Fainekos, G., Ito, H., & Kapinski, J. P. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500421 · Cited by 207 · see citing works
Self-citationSoftware and Hardware Testing Using Combinatorial Covering Suites
Hartman, A. (2005) · Springer eBooks · doi:10.1007/0-387-25036-0_10 · Cited by 193 · see citing works
Probabilistic Temporal Logic Falsification of Cyber-Physical Systems
Abbas, H., Fainekos, G., Sankaranarayanan, S., Ivančić, F., & Gupta, A. (2013) · ACM Transactions on Embedded Computing Systems · doi:10.1145/2465787.2465797 · Cited by 157 · see citing works
Self-citationAutomatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles
Althoff, M., & Lutz, S. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500374 · Cited by 142 · see citing works
Kapinski, J., Deshmukh, J. V., Jin, X., et al. (2016) · IEEE Control Systems · doi:10.1109/mcs.2016.2602089 · Cited by 140 · see citing works
Self-citationStellet, J. E., Zofka, M. R., Schumacher, J., Schamm, T., Niewels, F., & Zollner, J. M. (2015) · 2015 IEEE 18th International Conference on Intelligent Transportation Systems · doi:10.1109/itsc.2015.236 · Cited by 124 · see citing works
Traffic light recognition in varying illumination using deep learning and saliency map
John, V., Yoneda, K., Qi, B., et al. (2014) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2014.6958056 · Cited by 114 · see citing works
Compositional Falsification of Cyber-Physical Systems with Machine Learning Components
Dreossi, T., Donzé, A., & Seshia, S. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-57288-8_26 · Cited by 104 · see citing works
Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues
Chi, L., & Mu, Y. (2017) · arXiv · doi:10.48550/arxiv.1708.03798 · Cited by 87 · see citing works
Pedestrian detection with a Large-Field-Of-View deep network
Angelova, A., Krizhevsky, A., & Vanhoucke, V. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7139256 · Cited by 86 · see citing works
Utilizing S-TaLiRo as an automatic test generation framework for autonomous vehicles
Tuncali, C. E., Pavlic, T. P., & Fainekos, G. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795751 · Cited by 81 · see citing works
Self-citationDeep Predictive Models for Collision Risk Assessment in Autonomous Driving
Strickland, M., Fainekos, G., & Amor, H. B. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461160 · Cited by 78 · see citing works
Self-citationLearning and Verification of Feedback Control Systems using Feedforward Neural Networks
Dutta, S., Jha, S., Sankaranarayanan, S., & Tiwari, A. (2018) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2018.08.026 · Cited by 73 · see citing works
The worst-time-to-collision metric for situation identification
Wachenfeld, W., Junietz, P., Wenzel, R., et al. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535468 · Cited by 67 · see citing works
Safety verification of autonomous vehicles for coordinated evasive maneuvers
Althoff, M., Althoff, D., Wollherr, D., et al. (2010) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2010.5548121 · Cited by 66 · see citing works
Testing Autonomous Vehicle Software in the Virtual Prototyping Environment
Kim, B., Kashiba, Y., Dai, S., et al. (2016) · IEEE Embedded Systems Letters · doi:10.1109/les.2016.2644619 · Cited by 65 · see citing works
Counterexample-Guided Data Augmentation
Dreossi, T., Ghosh, S., Yue, X., Keutzer, K., Sangiovanni-Vincentelli, A., & Seshia, S. A. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/286 · Cited by 60 · see citing works
Rapidly-exploring Random Trees for Testing Automated Vehicles
Tuncali, C. E., & Fainekos, G. (2019) · IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917375 · Cited by 53 · see citing works
Self-citationSemantic Adversarial Deep Learning
Dreossi, T., Jha, S., & Seshia, S. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-96145-3_1 · Cited by 52 · see citing works
TrafficNet: An open naturalistic driving scenario library
Zhao, D. X., Guo, Y., & Jia, Y. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317860 · Cited by 49 · see citing works
Safe At Any Speed: A Simulation-Based Test Harness for Autonomous Vehicles
Abbas, H., O’Kelly, M., Rodionova, A., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17910-6_8 · Cited by 34 · see citing works
Robustness-guided temporal logic testing and verification for Stochastic Cyber-Physical Systems
Abbas, H., Hoxha, B., Fainekos, G., et al. (2014) · doi:10.1109/cyber.2014.6917426 · Cited by 33 · see citing works
Self-citationData-Driven Formal Reasoning and Their Applications in Safety Analysis of Vehicle Autonomy Features
Fan, C., Qi, B., & Mitra, S. (2018) · IEEE Design and Test · doi:10.1109/mdat.2018.2799804 · Cited by 32 · see citing works
VISPEC: A graphical tool for elicitation of MTL requirements
Hoxha, B., Mavridis, N., & Fainekos, G. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353863 · Cited by 30 · see citing works
Self-citationAPEX: Autonomous Vehicle Plan Verification and Execution
O’Kelly, M., Abbas, H., Gao, S., et al. (2016) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2016-01-0019 · Cited by 26 · see citing works
Computer-aided design for safe autonomous vehicles
O’Kelly, M., Abbas, H., & Mangharam, R. (2017) · Resilience Week (RWS) · doi:10.1109/rweek.2017.8088654 · Cited by 26 · see citing works
The SMT-based automatic road network generation in vehicle simulation environment
Kim, B., Jarandikar, A., Shum, J., et al. (2016) · International Conference on Embedded Software (EMSOFT) · doi:10.1145/2968478.2968498 · Cited by 21 · see citing works
Zofka, M. R., Essinger, M., Fleck, T., et al. (2018) · IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR) · doi:10.1109/simpar.2018.8376285 · Cited by 21 · see citing works
An efficient algorithm for monitoring practical TPTL specifications
Dokhanchi, A., Hoxha, B., Tuncali, C. E., et al. (2016) · ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE) · doi:10.1109/memcod.2016.7797763 · Cited by 12 · see citing works
Self-citationFunctional gradient descent optimization for automatic test case generation for vehicle controllers
Tuncali, C. E., Yaghoubi, S., Pavlic, T. P., et al. (2017) · doi:10.1109/coase.2017.8256245 · Cited by 11 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 11
- W. Burgard, U. Franke, M. Enzweiler, and M. Trivedi, "The Mobile Revolution -Machine Intelligence for Autonomous Vehicles (Dagstuhl Seminar 15462)," Dagstuhl Reports, vol. 5, no. 11, pp. 62-70, 2016. [Online]. Available: http://drops.dagstuhl.de/opus/volltexte/2016/5764
- D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network," in Advances in neural information processing systems, 1989, pp. 305-313.
- Y. Tian, K. Pei, S. Jana, and B. Ray, "Deeptest: Automated testing of deep-neural-network-driven autonomous cars," in 40th International Conference on Software Engineering (ICSE), 2018.
- S. M. Loos, A. Platzer, and L. Nistor, "Adaptive cruise control: Hybrid, distributed, and now formally verified," in Formal Methods, ser. LNCS, vol. 6664. Springer, 2011, pp. 42-56.
- R. Alur, Principles of Cyber-Physical Systems. MIT Press, 2015.
- B. Hoxha, H. Bach, H. Abbas, A. Dokhanchi, Y. Kobayashi, and G. Fainekos, "Towards formal specification visualization for testing and monitoring of cyber-physical systems," in International Workshop on Design and Implementation of Formal Tools and Systems, 2014.
- D. R. Kuhn, R. N. Kacker, and Y. Lei, Introduction to combinatorial testing. CRC press, 2013.
- R. Schubert, E. Richter, and G. Wanielik, "Comparison and evaluation of advanced motion models for vehicle tracking," in 2008 11th Interna- tional Conference on Information Fusion, June 2008, pp. 1-6.
- B. Grabe, T. Ike, and M. Hötter, "Evidence based evaluation method for grid-based environmental representation," in FUSION. IEEE, 2009, pp. 1234-1240.
- O. Michel, "Cyberbotics ltd. Webots: professional mobile robot simula- tion," International Journal of Advanced Robotic Systems, vol. 1, no. 1, p. 5, 2004.
- M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al., "A density-based algorithm for discovering clusters in large spatial databases with noise." in Kdd, vol. 96, no. 34, 1996, pp. 226-231.
Cited by
Sorted by citations
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
- Authors
- Tuncali, C. E., Fainekos, G., Prokhorov, D., Ito, H., & Kapinski, J.
- Affiliations
- Arizona State University (US)
- MathWorks (United Kingdom) (GB)
- Amazon (United States) (US)
- Venue
- IEEE Transactions on Intelligent Vehicles
- Published
- 2020-06
- Volume
- 5
- Issue
- 2
- Pages
- 265-280
- ISSN
- 2379-8858 (ISSN-L)
- 2379-8904
- OpenAlex
- W2965407852
- Cited by
- 129
- References
- 54
- Citations by year
- 2026: 8 · 2025: 14 · 2024: 21 · 2023: 31 · 2022: 18 · 2021: 18 · 2020: 16 · 2019: 3
- FWCI
- 33.04
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Intelligent Vehicles (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (CNS 1446730, CNS 1350420)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Formal Methods in Verification
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- autonomous vehicles, test case generation, signal temporal logic, deep neural network, covering arrays, sensor modeling, corner cases, control algorithm evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
