DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems
Open accessAbstract
While Deep Neural Networks (DNNs) have established the fundamentals of image-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To address the safety issues in autonomous driving systems, a recent set of testing techniques have been designed to automatically generate artificial driving scenes to enrich test suite, e.g., generating new input images transformed from the original ones. However, these techniques are insufficient due to two limitations: first, many such synthetic images often lack diversity of driving scenes, and hence compromise the resulting efficacy and reliability. Second, for machine-learning-based systems, a mismatch between training and application domain can dramatically degrade system accuracy, such that it is necessary to validate inputs for improving system robustness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Specific data anonymization methods have been designed for relational data, set-valued data, and image data.
“Specific DA methods have also been designed for relational data [262], set-valued data [320, 151], and image data [239, 97].”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [97]
The citation is attached to the image-data part of the sentence. Section 3.2.1 describes validating image inputs to autonomous-driving systems, and the Introduction describes generating driving-scene images for testing. Neither passage addresses data anonymization; searches for image anonymization and privacy found no relevant passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Paul Ammann and Jeff Offutt. 2016. Introduction to software testing. Cambridge University Press.
- Shane Barratt and Rishi Sharma. 2018. A Note on the Inception Score. arXiv preprint arXiv:1801.01973 (2018). · arXiv
- Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer. http://research.microsoft.com/en-us/um/people/cmbishop/prml/
- Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016). · arXiv
- Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis us- ing convolutional neural networks. In Advances in Neural Information Processing Systems. 262-270.
- Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on. IEEE, 2414-2423.
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672-2680.
- Divya Gopinath, Guy Katz, Corina S Pasareanu, and Clark Barrett. 2017. Deepsafe: A data-driven approach for checking adversarial robustness in neural networks. arXiv preprint arXiv:1710.00486 (2017). · arXiv
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real- time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694-711.
- Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. 2017. Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv:1703.05192 (2017). · arXiv
- Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013). · arXiv
- Nuo Li, Tao Xie, Maozhong Jin, and Chao Liu. 2010. Perturbation-based user- input-validation testing of web applications. Journal of Systems and Software 83, 11 (2010), 2263-2274.
- Peilun Li, Xiaodan Liang, Daoyuan Jia, and Eric P Xing. 2018. Semantic- aware Grad-GAN for Virtual-to-Real Urban Scene Adaption. arXiv preprint arXiv:1801.01726 (2018). · arXiv
- Ming-Yu Liu, Thomas Breuel, and Jan Kautz. 2017. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems. 700- 708.
- William M McKeeman. 1998. Differential testing for software. Digital Technical Journal 10, 1 (1998), 100-107.
- Christian Murphy, Gail E Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing. In SEKE, Vol. 8. 867-872.
- Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Au- tomated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 1-18.
- Dean A. Pomerleau. 1989. Advances in Neural Information Processing Systems 1. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, Chapter ALVINN: An Autonomous Land Vehicle in a Neural Network, 305-313.
- Haşim Sak, Andrew Senior, and Françoise Beaufays. 2014. Long short-term mem- ory recurrent neural network architectures for large scale acoustic modeling. In Fifteenth annual conference of the international speech communication association.
- Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. 2234-2242.
- S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortes. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering 42, 9 (Sept 2016), 805-824.
- Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 (2016), 805-824.
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). · arXiv
- Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. In Proceedings of the 40th International Conference on Software Engineering, Gothenburg, Sweden, May 27 -June 3, 2018 (ICSE 2018).
- Ian H Witten, Eibe Frank, Mark A Hall, and Christopher J Pal. 2016. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann.
- Xiaoyuan Xie, Joshua Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2009. Application of metamorphic testing to supervised classifiers. In Quality Software, 2009. QSIC'09. 9th International Conference on. IEEE, 135-144.
- Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84, 4 (2011), 544-558.
- Luona Yang, Xiaodan Liang, and Eric Xing. 2018. Unsupervised Real-to- Virtual Domain Unification for End-to-End Highway Driving. arXiv preprint arXiv:1801.03458 (2018). · arXiv
- Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised dual learning for image-to-image translation. arXiv preprint (2017).
- Zhi Quan Zhou, DH Huang, TH Tse, Zongyuan Yang, Haitao Huang, and TY Chen. 2004. Metamorphic testing and its applications. In Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004). 346-351.
Cited by
Sorted by citations
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612 · see citing works
“For example, in the field of autonomous driving, besides developing a perceptual system working in sunny scenes, academia and the industry are using numerous development and testing strategies to enhance the perceptual performance of the vehicles in nighttime/rainy scenes to guarantee the system’s reliability under a…”
BackgroundMethodDeng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326 · see citing works
Self-citationTesting machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251 · see citing works
Deep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · see citing works
An extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · see citing works
Self-citationA Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137 · see citing works
Free lunch for testing: fuzzing deep-learning libraries from open source
Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106 · see citing works
Self-citationProblems and opportunities in training deep learning software systems: an analysis of variance
Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416545 · Cited by 100 · see citing works
Fuzzing deep-learning libraries via automated relational API inference
Deng, Y., Yang, C., Wei, A., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549085 · Cited by 86 · see citing works
Self-citationMuffin: testing deep learning libraries via neural architecture fuzzing
Gu, J., Luo, X., Zhou, Y., & Wang, X. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510092 · Cited by 71 · see citing works
Rizvi, S. K. J., Azad, M. A., & Fraz, M. M. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09543-4 · Cited by 37 · see citing works
Ren, R., Wang, Y., Liu, F., Li, Z., & Xie, G. (2023) · 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS) · doi:10.1109/iwqos57198.2023.10188773 · Cited by 16 · see citing works
Chen, C., Wei, L., Zhang, L., Peng, Y., & Ning, J. (2022) · Security and Communication Networks · doi:10.1155/2022/2985308 · Cited by 8 · see citing works
Self-citation
- Authors
- Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S.
- Affiliations
- The University of Texas at Austin (US)
- Southern University of Science and Technology (CN)
- The University of Texas at Dallas (US)
- University of Texas at Austin, USA
- Venue
- Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2018-09-03
- Pages
- 132-142
- OpenAlex
- W2888307014
- Cited by
- 611
- References
- 3
- Citations by year
- 2026: 47 · 2025: 73 · 2024: 89 · 2023: 95 · 2022: 93 · 2021: 98 · 2020: 82 · 2019: 30 · 2018: 4
- FWCI
- 36.25
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- Funders
- National Science Foundation (CNS-1527727, CCF-1566589, CNS-CAREER-1750263, CCF-1704790)
- Shenzhen Peacock Plan (KQTD201611,2514355531)
- Science and Technology Innovation Committee Foundation of Shenzhen (ZDSYS201703031748284,JCYJ20170817110848086)
- Ministry of Science and Technology of the People's Republic of China (2017YFC0804002)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- metamorphic testing, input validation, autonomous driving systems, generative adversarial networks, deep neural network, system robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
