The DOI matches its Crossref record; 202 of the 202 references it cites were found in a registry or an index.
Software Engineering for AI-Based Systems: A Survey
Open accessAbstract
AI-based systems are software systems with functionalities enabled by at least one AI component (e.g., for image-, speech-recognition, and autonomous driving). AI-based systems are becoming pervasive in society due to advances in AI. However, there is limited synthesized knowledge on Software Engineering (SE) approaches for building, operating, and maintaining AI-based systems. To collect and analyze state-of-the-art knowledge about SE for AI-based systems, we conducted a systematic mapping study. We considered 248 studies published between January 2010 and March 2020. SE for AI-based systems is an emerging research area, where more than 2/3 of the studies have been published since 2018. The most studied properties of AI-based systems are dependability and safety. We identified multiple SE approaches for AI-based systems, which we classified according to the SWEBOK areas. Studies related to software testing and software quality are very prevalent, while areas like software maintenance seem neglected. Data-related issues are the most recurrent challenges. Our results are valuable for: researchers, to quickly understand the state-of-the-art and learn which topics need more research; practitioners, to learn about the approaches and challenges that SE entails for AI-based systems; and, educators, to bridge the gap among SE and AI in their curricula.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,583
(2004) · Cited by 5,198
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 3×
“Therefore, we decided to execute a hybrid search strategy [139], applying both a search string in Scopus3 and a snowballing strategy [209].”
MethodSystematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 2×
“This SMS has five main steps [111, 156] described in the following subsections.”
BackgroundGuidelines for conducting systematic mapping studies in software engineering: An update
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2015.03.007 · Cited by 2,669
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007) · Journal of Systems and Software · doi:10.1016/j.jss.2006.07.009 · Cited by 2,394
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245 · cited 5×
“Furthermore, five articles propose ML quality certification and assessment to mitigate quality risks of deployed AI systems [12, 94, 145, 163, 224].”
BackgroundInfluentialThe Goal Question Metric Approach
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994) · Cited by 1,651
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,135
Recommended Steps for Thematic Synthesis in Software Engineering
Cruzes, D. S., & Dyba, T. (2011) · 2011 International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.36 · Cited by 960
(2015) · neural information processing systems · Cited by 912
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Safety Verification of Deep Neural Networks
Huang, X., Kwiatkowska, M., Wang, S., & Wu, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_1 · Cited by 818
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Encyclopedia of software engineering
(1994) · Choice Reviews Online · doi:10.5860/choice.32-0639 · Cited by 707
Challenges in autonomous vehicle testing and validation
Koopman, P., & Wagner, M. (2016) · SAE International Journal of Transportation Safety · doi:10.4271/2016-01-0128 · Cited by 675
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Robust Physical-World Attacks on Deep Learning Models
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08945 · Cited by 444
FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
Klampfl, L., & Wotawa, F. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.10 · Cited by 431
DeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426
Clipper: A Low-Latency Online Prediction Serving System
Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., & Stoica, I. (2016) · arXiv · doi:10.48550/arxiv.1612.03079 · Cited by 415
TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
Baylor, D., Breck, E., Cheng, H.-T., Fiedel, N., Foo, C. Y., Haque, Z., Haykal, S., Ispir, M., Jain, V., Koc, L., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098021 · Cited by 350
Structural Test Coverage Criteria for Deep Neural Networks
Sun, Y., Huang, X., Kroening, D., Sharp, J. J., Hill, M., & Ashmore, R. (2019) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3358233 · Cited by 348
Concolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332
The ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310
Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305
A comprehensive study on deep learning bug characteristics
Islam, M. J., Nguyen, G., Pan, R., & Rajan, H. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338955 · Cited by 305
An empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213866 · Cited by 304
Standardizing Ethical Design for Artificial Intelligence and Autonomous Systems
Bryson, J., & Winfield, A. (2017) · Computer · doi:10.1109/mc.2017.154 · Cited by 293
A method for evaluating rigor and industrial relevance of technology evaluations
Ivarsson, M., & Gorschek, T. (2011) · Empirical Software Engineering · doi:10.1007/s10664-010-9146-4 · Cited by 279
Formal Security Analysis of Neural Networks using Symbolic Intervals
Wang, S., Pei, K., Whitehouse, J., Yang, J., & Jana, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10829 · Cited by 265
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
Saunders, W., Sastry, G., Stuhlmüller, A., & Evans, O. (2018) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/mzmg4429 · Cited by 265
Testing vision-based control systems using learnable evolutionary algorithms
Abdessalem, R. B., Nejati, S., Briand, L. C., & Stifter, T. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180160 · Cited by 262
An Abstraction-Refinement Approach to Verification of Artificial Neural Networks
Pulina, L., & Tacchella, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-14295-6_24 · Cited by 261
Testing machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241 · cited 2×
“[125] 72 studies (1998-2018) adaptability, scalability, safety, privacy Software construction.”
BackgroundAutomatically testing self-driving cars with search-based procedural content generation
Gambi, A., Mueller, M., & Fraser, G. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330566 · Cited by 226
Requirements Engineering for Machine Learning: Perspectives from Data Scientists
Vogelsang, A., & Borg, M. (2019) · 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew.2019.00050 · Cited by 223
DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks
Feng, Y., Shi, Q., Gao, X., Wan, J., Fang, C., & Chen, Z. (2020) · Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3395363.3397357 · Cited by 217
Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., & Crnkovic, I. (2019) · Lecture Notes in Business Information Processing · doi:10.1007/978-3-030-19034-7_14 · Cited by 214
The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214
On the safety of machine learning: Cyber-physical systems, decision sciences, and data products
Varshney, K. R., & Alemzadeh, H. (2017) · Big Data · doi:10.1089/big.2016.0051 · Cited by 211 · cited 3×
“We can highlight that in the 11 primary studies discussing roadmaps, they are often related to safety and the standard ISO 26262 [7, 66, 72, 105, 201].”
BackgroundMODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211
Human model evaluation in interactive supervised learning
Fiebrink, R., Cook, P. R., & Trueman, D. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978965 · Cited by 207
On the reliability of mapping studies in software engineering
Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I., & de Almeida, E. S. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.04.076 · Cited by 206
VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems
Dreossi, T., Fremont, D. J., Ghosh, S., Kim, E., Ravanbakhsh, H., Vazquez-Chanlatte, M., & Seshia, S. A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-25540-4_25 · Cited by 200
Data scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
TensorFuzz: Debugging Neural Networks with Coverage-Guided FuzzingPreprint
Odena, A., & Goodfellow, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10875 · Cited by 198
On testing machine learning programs
Braiek, H. B., & Khomh, F. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110542 · Cited by 197
Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World
Fremont, D. J., Kim, E., Pant, Y. V., Seshia, S. A., Acharya, A., Bruso, X., Wells, P., Lemke, S., Lu, Q., & Mehta, S. (2020) · 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294368 · Cited by 197
Automated directed fairness testing
Udeshi, S., Arora, P., & Chattopadhyay, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238165 · Cited by 193
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Dwarakanath, A., Ahuja, M., Sikand, S., Rao, R. M., Bose, R. P. J. C., Dubash, N., & Podder, S. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213858 · Cited by 183
Data Validation for Machine Learning
Breck, E. J., Zinkevich, M., Polyzotis, N., Whang, S. E., & Roy, S. (2019) · Cited by 182
How does Machine Learning Change Software Development Practices?
Wan, Z., Xia, X., Lo, D., & Murphy, G. C. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2937083 · Cited by 181
Deng, L. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2762725 · Cited by 180
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
Black box fairness testing of machine learning models
Aggarwal, A., Lohia, P., Nagar, S., Dey, K., & Saha, D. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338937 · Cited by 173
Ethical Challenges in Data-Driven Dialogue Systems
Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., & Pineau, J. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278777 · Cited by 168
Model-Reuse Attacks on Deep Learning Systems
Ji, Y., Zhang, X., Ji, S., Luo, X., & Wang, T. (2018) · Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3243734.3243757 · Cited by 167
Non-Functional Requirements for Machine Learning: Challenges and New Directions
Horkoff, J. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00050 · Cited by 165
An Empirical Study of Common Challenges in Developing Deep Learning Applications
Zhang, T., Gao, C., Ma, L., Lyu, M. R.-T., & Kim, M. (2019) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2019.00020 · Cited by 165
DeepStellar: model-based quantitative analysis of stateful deep learning systems
Du, X., Xie, X., Li, Y., Ma, L., Liu, Y., & Zhao, J. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338954 · Cited by 163
On the pragmatic design of literature studies in software engineering: an experience-based guideline
Kuhrmann, M., Fernández, D. M., & Daneva, M. (2017) · Empirical Software Engineering · doi:10.1007/s10664-016-9492-y · Cited by 161
Borg, M., Englund, C., Wnuk, K., Durán, B., Levandowski, C., Gao, S., Tan, Y., Kaijser, H., Lönn, H., & Törnqvist, J. (2019) · Journal of Automotive Software Engineering · doi:10.2991/jase.d.190131.001 · Cited by 159
DeepBillboard: systematic physical-world testing of autonomous driving systems
Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380422 · Cited by 157
An Empirical Study of Bugs in Machine Learning Systems
Thung, F., Wang, S., Lo, D., & Jiang, L. (2012) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2012.22 · Cited by 149
Making the Case for Safety of Machine Learning in Highly Automated Driving
Burton, S., Gauerhof, L., & Heinzemann, C. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66284-8_1 · Cited by 143
Towards Verified Artificial Intelligence
Seshia, S. A., Sadigh, D., & Sastry, S. S. (2016) · arXiv · doi:10.48550/arxiv.1606.08514 · Cited by 142
Guo, Q., Chen, S., Xie, X., Ma, L., Hu, Q., Liu, H., Liu, Y., Zhao, J., & Li, X. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00080 · Cited by 141
An Analysis of ISO 26262: Using Machine Learning Safely in Automotive SoftwarePreprint
Salay, R., Queiroz, R., & Czarnecki, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02435 · Cited by 140
Data Management Challenges for Deep Learning
Munappy, A. R., Bosch, J., Olsson, H. H., Arpteg, A., & Brinne, B. (2019) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2019.00030 · Cited by 140
On Challenges in Machine Learning Model Management
Schelter, S., Bießmann, F., Januschowski, T., Salinas, D., Seufert, S., & Szarvas, G. (2015) · IEEE Data(base) Engineering Bulletin · Cited by 139
Engineering safety in machine learning
Varshney, K. R. (2016) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2016.7888195 · Cited by 138
Requirements-driven test generation for autonomous vehicles with machine learning components
Tuncali, C. E., Fainekos, G., Prokhorov, D., Ito, H., & Kapinski, J. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2955903 · Cited by 129
Perkusich, M., Silva, L. C. E., Costa, A., Ramos, F., Saraiva, R., Freire, A., Dilorenzo, E., Dantas, E., Santos, D. F. S., Gorgônio, K., et al. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106241 · Cited by 128
Adoption and Effects of Software Engineering Best Practices in Machine Learning
Serban, A., van der Blom, K., Hoos, H., & Visser, J. (2020) · Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1145/3382494.3410681 · Cited by 127
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126
Toward a Framework for Highly Automated Vehicle Safety Validation
Koopman, P. J., & Wagner, M. (2018) · SAE technical papers on CD-ROM/SAE technical paper series · doi:10.4271/2018-01-1071 · Cited by 126
Benchmarking Kappa: Interrater Agreement in Software Process Assessments
Emam, K. E. (1999) · Empirical Software Engineering · doi:10.1023/a:1009820201126 · Cited by 120
Machine Learning in Artificial Intelligence: Towards a Common Understanding
Kühl, N., Goutier, M., Hirt, R., & Satzger, G. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.630 · Cited by 116
Towards Corner Case Detection for Autonomous Driving
Bolte, J.-A., Bär, A., Lipinski, D., & Fingscheidt, T. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8813817 · Cited by 116
Trials and tribulations of developers of intelligent systems: A field study
Hill, C., Bellamy, R., Erickson, T. D., & Burnett, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739680 · Cited by 115
Open source software ecosystems: A Systematic mapping
Bedoya, Ó. H. F., Ameller, D., Costal, D., & Franch, X. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.07.007 · Cited by 113
Self-citationRequirements Engineering Challenges in Building AI-Based Complex Systems
Belani, H., Vuković, M., & Car, Ž. (2019) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew.2019.00051 · Cited by 113
Ishikawa, F., & Yoshioka, N. (2019) · doi:10.1109/cesser-ip.2019.00009 · Cited by 109
A Cloud-Based Framework for Machine Learning Workloads and Applications
Lopez Garcia, A., De Lucas, J. M., Antonacci, M., Zu Castell, W., David, M., Hardt, M., Lloret Iglesias, L., Molto, G., Plociennik, M., Tran, V., et al. (2020) · IEEE Access · doi:10.1109/access.2020.2964386 · Cited by 108
Did We Test All Scenarios for Automated and Autonomous Driving Systems?
Hauer, F., Schmidt, T., Holzmüller, B., & Pretschner, A. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917326 · Cited by 106
ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI
Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., & Oum, P. (2019) · 2019 IEEE International Conference on Cloud Engineering (IC2E) · doi:10.1109/ic2e.2019.00025 · Cited by 105
Liu, D. C., Rogers, S., Shiau, R., Kislyuk, D., Kevin, C., Zhong, Z., Liu, J., & Jing, Y. (2017) · International World Wide Web Conference (WWW) · doi:10.1145/3041021.3054202 · Cited by 103
SOTER: A Runtime Assurance Framework for Programming Safe Robotics Systems
Desai, A., Ghosh, S., Seshia, S. A., Shankar, N., & Tiwari, A. (2019) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2019.00027 · Cited by 102
Behutiye, W., Karhapää, P., López, L., Illa, X. B., Martínez‐Fernández, S., Vollmer, A. M., Rodríguez, P., Franch, X., & Oivo, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106225 · Cited by 101
Self-citationApplying Deep Learning to Airbnb Search
Haldar, M., Abdool, M., Ramanathan, P., Xu, T., Yang, S.-L., Duan, H., Zhang, Q., Barrow-Williams, N., Turnbull, B., Collins, B. M., et al. (2018) · Knowledge Discovery and Data Mining · doi:10.1145/3292500.3330658 · Cited by 100
What Is Really Different in Engineering AI-Enabled Systems?
Özkaya, İ. (2020) · IEEE Software · doi:10.1109/ms.2020.2993662 · Cited by 100
Studying Software Engineering Patterns for Designing Machine Learning Systems
Washizaki, H., Uchida, H., Khomh, F., & Guéhéneuc, Y. (2019) · International Workshop on Empirical Software Engineering in Practice (IWESEP) · doi:10.1109/iwesep49350.2019.00017 · Cited by 96
Empirical Standards for Software Engineering ResearchPreprint
Ralph, P., bin Ali, N., Baltes, S., Bianculli, D., Díaz, J., Dittrich, Y., Ernst, N., Felderer, M., FELDT, R. H., Filieri, A., et al. (2020) · arXiv (Cornell University) · doi:10.4230/lipics.esem.2026.10 · Cited by 96
Software engineering for machine-learning applications: The road ahead
Khomh, F., Adams, B., Cheng, J., Fokaefs, M., & Antoniol, G. (2018) · IEEE Software · doi:10.1109/ms.2018.3571224 · Cited by 95
150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
Bernardi, L., Mavridis, T., & Estevez, P. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330744 · Cited by 95
Uncertainty in Machine Learning: A Safety Perspective on Autonomous Driving
Shafaei, S., Kugele, S., Osman, M. H., & Knoll, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99229-7_39 · Cited by 93
DeepFault: Fault Localization for Deep Neural Networks
Enişer, H. F., Gerasimou, S., & Şen, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-16722-6_10 · Cited by 92
Nascimento, E., Nguyen‐Duc, A., Sundbø, I., & Conte, T. U. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03751 · Cited by 90
Understanding Development Process of Machine Learning Systems: Challenges and Solutions
Nascimento, E., Ahmed, I., de Oliveira, E. C. C., Palheta, M. P., Steinmacher, I., & Conte, T. U. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870157 · Cited by 89
Engineering problems in machine learning systems
Kuwajima, H., Yasuoka, H., & Nakae, T. (2020) · Machine Learning · doi:10.1007/s10994-020-05872-w · Cited by 89
Input Prioritization for Testing Neural Networks
Byun, T., Sharma, V., Vijayakumar, A., Rayadurgam, S., & Cofer, D. D. (2019) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest.2019.000-6 · Cited by 85
DiffChaser: Detecting Disagreements for Deep Neural Networks
Xie, X., Ma, L., Wang, H., Li, Y., Liu, Y., & Li, X. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/800 · Cited by 83
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81
DeepConcolic: Testing and Debugging Deep Neural Networks
Sun, Y., Huang, X., Kroening, D., Sharp, J. J., Hill, M., & Ashmore, R. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00051 · Cited by 74
An Empirical Study on Real Bugs for Machine Learning Programs
Sun, X., Zhou, T., Li, G., Hu, J., Yang, H. M., & Li, B. (2017) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.41 · Cited by 72
A Survey of Software Quality for Machine Learning Applications
Masuda, S., Ono, K., Yasue, T., & Hosokawa, N. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00061 · Cited by 72
Research Directions for Engineering Big Data Analytics Software
Otero, C. E., & Peter, A. M. (2014) · IEEE Intelligent Systems · doi:10.1109/mis.2014.76 · Cited by 70
Trustworthy AI in the Age of Pervasive Computing and Big Data
Kumar, A., Braud, T., Tarkoma, S., & Hui, P. (2020) · 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops48775.2020.9156127 · Cited by 68
Using Ontologies for Test Suites Generation for Automated and Autonomous Driving Functions
Klück, F., Li, Y., Nica, M., Tao, J., & Wotawa, F. (2018) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2018.00-20 · Cited by 68
Verification of Binarized Neural Networks via Inter-neuron Factoring
Cheng, C., Nührenberg, G., Huang, C.-H., & Rueß, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03592-1_16 · Cited by 60
Semantic Adversarial Deep Learning
Dreossi, T., Jha, S., & Seshia, S. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-96145-3_1 · Cited by 60
Toward Requirements Specification for Machine-Learned Components
Rahimi, M., Guo, J., Kokaly, S., & Chećhik, M. (2019) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew.2019.00049 · Cited by 56
Teaching software engineering for AI-enabled systems
Kästner, C., & Kang, E. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3377814.3381714 · Cited by 53
Structuring Validation Targets of a Machine Learning Function Applied to Automated Driving
Gauerhof, L., Munk, P., & Burton, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99130-6_4 · Cited by 52
Defining Explainable AI for Requirements Analysis
Sheh, R., & Monteath, I. (2018) · Künstliche Intell. · doi:10.1007/s13218-018-0559-3 · Cited by 52
Artificial intelligence poised to ride a new wave
Anthes, G. (2017) · Communications of the ACM · doi:10.1145/3088342 · Cited by 50
Combinatorial Testing for Deep Learning SystemsPreprint
Ma, L., Zhang, F., Xue, M., Li, B., Liu, Y., Zhao, J., & Wang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07723 · Cited by 50
Adapting SQuaRE for Quality Assessment of Artificial Intelligence Systems
Kuwajima, H., & Ishikawa, F. (2019) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2019.00035 · Cited by 50
Detecting Adversarial Samples for Deep Neural Networks through Mutation TestingPreprint
Wang, J., Sun, J., Zhang, P., & Wang, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.05010 · Cited by 49
Towards a holistic software systems engineering approach for dependable autonomous systems
Aniculăesei, A., Grieser, J., Rausch, A., et al. (2018) · doi:10.1145/3194085.3194091 · Cited by 49
Experimental Resilience Assessment of an Open-Source Driving Agent
Rubaiyat, A. H. M., Qin, Y., & Alemzadeh, H. (2018) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc.2018.00016 · Cited by 49
Molina, C., de Almeida, J. R., Vismari, L. F., Gonzalez, R. I. R., Naufal, J. K., & Camargo, J. B. (2017) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w.2017.14 · Cited by 47
Deep Validation: Toward Detecting Real-World Corner Cases for Deep Neural Networks
Wu, W., Xu, H. F., Zhong, S., Lyu, M. R.-T., & King, I. (2019) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2019.00026 · Cited by 47
Automotive safety and machine learning
Henriksson, J., Borg, M., & Englund, C. (2018) · doi:10.1145/3194085.3194090 · Cited by 43
Menzies, T. (2020) · IEEE Software · doi:10.1109/ms.2019.2954841 · Cited by 43
Machine Learning Software Engineering in Practice: An Industrial Case StudyPreprint
Rahman, M. S., Rivera, E., Khomh, F., Guéhéneuc, Y., & Lehnert, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.07154 · Cited by 40
Bottester: Testing Conversational Systems with Simulated Users
Vasconcelos, M., Candello, H. C. D. S. P., Pinhanez, C. S., & dos Santos, T. D. (2017) · Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3160504.3160584 · Cited by 38
Packaging and Sharing Machine Learning Models via the Acumos AI Open Platform
Zhao, S., Talasila, M., Jacobson, G., Borcea, C., Aftab, S. A., & Murray, J. F. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00135 · Cited by 38
Paracosm: A Language and Tool for Testing Autonomous Driving SystemsPreprint
Majumdar, R., Mathur, A. S., Pirron, M., Stegner, L., & Damien Zufferey, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01084 · Cited by 38
Recent Trends in Formal Validation and Verification of Autonomous Robots Software
Ingrand, F. (2019) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc.2019.00059 · Cited by 38
Efficient Autonomy Validation in Simulation with Adaptive Stress Testing
Koren, M., & Kochenderfer, M. J. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917403 · Cited by 38
The Role of Design in Creating Machine-Learning-Enhanced User Experience.
Yang, Q. (2017) · National Conference on Artificial Intelligence · Cited by 32
Invited Paper: What is AI Software Testing? and Why
Gao, J. Z., Tao, C., Jie, D., & Lu, S. (2019) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2019.00015 · Cited by 31
Towards a Framework for Safety Assurance of Autonomous Systems
McDermid, J. A., Jia, Y., & Habli, I. (2019) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 30
Architecting AI Deployment: A Systematic Review of State-of-the-Art and State-of-Practice Literature
John, M. M., Olsson, H. H., & Bosch, J. (2021) · Lecture notes in business information processing · doi:10.1007/978-3-030-67292-8_2 · Cited by 30
A bibliometric assessment of software engineering themes, scholars and institutions (2013–2020)
Wong, W. E., Mittas, N., Arvanitou, E.-M., & Li, Y. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111029 · Cited by 30
Hains, G., Jakobsson, A., & Khmelevsky, Y. (2018) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon.2018.8369576 · Cited by 29
Versioning for End-to-End Machine Learning Pipelines
van der Weide, T., Papadopoulos, D., Smirnov, O., Zielinski, M., & van Kasteren, T. L. M. (2017) · Workshop on Data Management for End-to-End Machine Learning (DEEM) · doi:10.1145/3076246.3076248 · Cited by 28
Tuncali, C. E., Fainekos, G., Ito, H., & Kapinski, J. P. (2018) · International Conference on Hybrid Systems: computation and control · doi:10.1145/3178126.3187004 · Cited by 28
Model-based design for CPS with learning-enabled components
Hartsell, C., Mahadevan, N., Ramakrishna, S., Dubey, A. K., Bapty, T. A., Johnson, T. T., Koutsoukos, X., Sztipanovits, J., & Karsai, G. (2019) · IEEE Workshop on Design Automation for CPS and IoT (DESTION) · doi:10.1145/3313151.3313166 · Cited by 28
Software Engineering Practice in the Development of Deep Learning ApplicationsPreprint
Zhang, X., Yang, Y., Feng, Y., & Chen, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03156 · Cited by 28
Gharib, M., Lollini, P., Botta, M., Amparore, E. G., Donatelli, S., & Bondavalli, A. (2018) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w.2018.00074 · Cited by 26
Designing Safety Critical Software Systems to Manage Inherent Uncertainty
Serban, A. C. (2019) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c.2019.00051 · Cited by 25
Testing Untestable Neural Machine Translation: An Industrial Case
Zheng, W., Wang, W., Liu, D., Zhang, C., Zeng, Q., Deng, Y., Yang, W., He, P., & Xie, T. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00131 · Cited by 25
An Architecture for Agile Machine Learning in Real-Time Applications
Schleier-Smith, J. (2015) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2788628 · Cited by 24
Božić, J., & Wotawa, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99927-2_3 · Cited by 24
A Conceptual Safety Supervisor Definition and Evaluation Framework for Autonomous Systems
Feth, P., Schneider, D., & Adler, R. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66266-4_9 · Cited by 23
Intelligent Software Engineering: Synergy Between AI and Software Engineering
Xie, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99933-3_1 · Cited by 23
Importance-driven deep learning system testing
Gerasimou, S., Enişer, H. F., Şen, A., & Çakan, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377812.3390793 · Cited by 22
Observation Based Creation of Minimal Test Suites for Autonomous Vehicles
Wolschke, C., Kuhn, T. S., Rombach, D., & Liggesmeyer, P. (2017) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2017.46 · Cited by 21
A Test Architecture for Machine Learning Product
Nishi, Y., Masuda, S., Ogawa, H., & Uetsuki, K. (2018) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2018.00060 · Cited by 21
MULDEF: Multi-model-based Defense Against Adversarial Examples for Neural NetworksPreprint
Srisakaokul, S., Zhang, Y., Zhong, Z., Yang, W. P., Xie, T., & Li, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00065 · Cited by 21
[Invited] Quality Assurance of Machine Learning Software
Nakajima, S. (2018) · IEEE Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce.2018.8574766 · Cited by 21
Does AI have a methodology which is different from software engineering?
Partridge, D., & Wilks, Y. A. (1987) · Artificial Intelligence Review · doi:10.1007/bf00130012 · Cited by 19
Concepts in Quality Assessment for Machine Learning - From Test Data to Arguments
Ishikawa, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00847-5_39 · Cited by 19
Coates, D. L., & Martin, A. J. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2915062 · Cited by 19
Improving ML Safety with Partial Specifications
Salay, R., & Czarnecki, K. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26250-1_23 · Cited by 19
Challenges of Testing Machine Learning Applications
Huang, S. (2018) · International Journal of Performability Engineering · doi:10.23940/ijpe.18.06.p18.12751282 · Cited by 18
Lowering the barrier to applying machine learning
Patel, K. (2010) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1866218.1866222 · Cited by 17
Characterizing machine learning process: A maturity frameworkPreprint
Akkiraju, R., Sinha, V. S., Xu, A., Mahmud, J., Gundecha, P., Liu, Z., Liu, X., & Schumacher, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04871 · Cited by 17
Dataset Diversity for Metamorphic Testing of Machine Learning Software
Nakajima, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-13651-2_2 · Cited by 17
Solution Patterns for Machine Learning
Nalchigar, S., Yu, E. S. K., Obeidi, Y., Carbajales, S., Green, J., & Chan, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-21290-2_39 · Cited by 17
Counterexample-Guided Synthesis of Perception Models and Control
Ghosh, S., Pant, Y. V., Ravanbakhsh, H., & Seshia, S. A. (2021) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc50511.2021.9482896 · Cited by 17
On the Diversity of Machine Learning Models for System Reliability
Machida, F. (2019) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc47002.2019.00058 · Cited by 16
Sensemaking Practices in the Everyday Work of AI/ML Software Engineering
Wolf, C. T., & Paine, D. (2020) · doi:10.1145/3387940.3391496 · Cited by 16
Beyond the technical challenges for deploying Machine Learning solutions in a software companyPreprint
Flaounas, I. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02363 · Cited by 15
Safety assessment of automated vehicle functions by simulation-based fault injection
Juez, G., Amparan, E., Lattarulo, R. A., Perez, J., Ruíz, A., & Espinoza, H. (2017) · IEEE International Conference on Vehicular Electronics and Safety (ICVES) · doi:10.1109/icves.2017.7991928 · Cited by 14
Continuous Argument Engineering: Tackling Uncertainty in Machine Learning Based Systems
Ishikawa, F., & Matsuno, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99229-7_2 · Cited by 14
Jentzsch, S. F., & Hochgeschwender, N. (2019) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew.2019.00025 · Cited by 13
A Quantitative Analysis Framework for Recurrent Neural Network
Du, X., Xie, X., Li, Y., Ma, L., Liu, Y., & Zhao, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00102 · Cited by 12
How Tertiary Studies perform Quality Assessment of Secondary Studies in Software EngineeringPreprint
Costal, D., Farré, C., Franch, X., & Quer, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03820 · Cited by 12
Self-citationRepresentative Safety Assessment of Autonomous Vehicle for Public Transportation
Adedjouma, M., Pedroza, G., & Bannour, B. (2018) · IEEE International Symposium on Real-Time Distributed Computing (ISORC) · doi:10.1109/isorc.2018.00025 · Cited by 11
Requirements Assurance in Machine Learning.
Banks, A., & Ashmore, R. (2019) · National Conference on Artificial Intelligence · Cited by 11
Reusability in Artificial Neural Networks: An Empirical Study
Ghofrani, J., Kozegar, E., Bozorgmehr, A., & Soorati, M. D. (2019) · International Software Product Line Conference (SPLC) · doi:10.1145/3307630.3342419 · Cited by 11
Reliability Validation of Learning Enabled Vehicle Tracking
Sun, Y., Zhou, Y., Maskell, S. R., Sharp, J. J., & Huang, X. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196932 · Cited by 11
(2016) · Cited by 10
An AI Software Test Method Based on Scene Deductive Approach
Zhao, X., & Gao, X. (2018) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2018.00017 · Cited by 9
Design Automation for Intelligent Automotive Systems
Lan, S., Huang, C., Wang, Z., Liang, H., Su, W., & Zhu, Q. (2018) · IEEE International Test Conference (ITC) · doi:10.1109/test.2018.8624723 · Cited by 9
Safe-by-Design Development Method for Artificial Intelligent Based Systems
Pedroza, G., & Adedjouma, M. (2019) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2019-094 · Cited by 9
Synergy between Machine/Deep Learning and Software Engineering: How Far Are We?Preprint
Wang, S., Huang, L., Ge, J., Zhang, T., Feng, H., Li, M., Zhang, H., & Ng, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05515 · Cited by 9
Improving Certified Robustness Via Statistical Learning with Logical Reasoning
Yang, Z., Zhao, Z., Wang, B., Zhang, J., Liu, J., Pei, H., Karlaš, B., Liu, J., Guo, H., Zhang, C., et al. (2022) · neural information processing systems · doi:10.52202/068431-2526 · Cited by 9
Keeping intelligence under control
Mallozzi, P., Pelliccione, P., & Menghi, C. (2018) · doi:10.1145/3195555.3195558 · Cited by 8
Sarathy, P., Baruah, S., Cook, S. P., & Wolf, M. (2019) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc43569.2019.9081649 · Cited by 8
Towards Testing of Deep Learning Systems with Training Set ReductionPreprint
Spieker, H., & Arnaud Gotlieb, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04169 · Cited by 7
An Empirical Study of Software Architecture for Machine Learning.Preprint
Serban, A., & Visser, J. (2021) · arXiv (Cornell University) · Cited by 7
Telemade: A Testing Framework for Learning-Based Malware Detection Systems
Yang, W. M., & Xie, T. (2018) · National Conference on Artificial Intelligence · Cited by 6
Martínez‐Fernández, S., Franch, X., Jedlitschka, A., Oriol, M., & Trendowicz, A. (2020) · arXiv (Cornell University) · Cited by 6
Self-citationLearning with Safety Requirements: State of the Art and Open Questions.
Leofante, F., Pulina, L., & Tacchella, A. (2016) · UnissResearch (Università degli Studi di Sassari) · Cited by 3
How do implementation bugs affect the results of machine learning algorithms?
Leotta, M., Olianas, D., Ricca, F., & Noceti, N. (2019) · ACM Symposium on Applied Computing · doi:10.1145/3297280.3297411 · Cited by 3
Colomo‐Palacios, R. (2019) · Springer proceedings in complexity · doi:10.1007/978-3-030-30809-4_31 · Cited by 3
Control and Safety of Autonomous Vehicles with Learning-Enabled Components
Bansal, S., & Tomlin, C. J. (2018) · Unmanned system technologies · doi:10.1007/978-3-319-97301-2_4 · Cited by 2
Cited by
Perez-Cerrolaza, J., Abella, J., Borg, M., Donzella, C., Cerquides, J., Cazorla, F. J., Englund, C., Tauber, M., Nikolakopoulos, G., & Flores, J. L. (2024) · ACM Computing Surveys · doi:10.1145/3626314 · Cited by 185
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Steidl, M., Felderer, M., & Ramler, R. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111615 · Cited by 126
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Metadata
- Authors
- Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S.
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2022-04
- Volume
- 31
- Issue
- 2
- Pages
- 1-59
- ISSN
- 2331-8422 (ISSN-L)
- 1049-331X
- 1557-7392
- OpenAlex
- W4220891756
- Cited by
- 335
- References
- 202
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.