The Future of AI-Driven Software Engineering
Open accessThe DOI matches its Crossref record; 112 of the 112 references it cites were found in a registry or an index.
Abstract
A paradigm shift is underway in Software Engineering, with AI systems such as LLMs playing an increasingly important role in boosting software development productivity. This trend is anticipated to persist. In the next years, we expect a growing symbiotic partnership between human software developers and AI. The Software Engineering research community cannot afford to overlook this trend; we must address the key research challenges posed by the integration of AI into the software development process. In this article, we present our vision of the future of software development in an AI-driven world and explore the key challenges that our research community should address to realize this vision.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This transformation changes the traditional software development paradigms, not only in tooling and automation, but also in developer collaboration and decision-making [160, 189].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [189]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Explainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.37 · Cited by 863
Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges
Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., & Zhu, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32236-6_51 · Cited by 758
Metamorphic Testing: A Review of Challenges and Opportunities
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2019) · ACM Computing Surveys · doi:10.1145/3143561 · Cited by 665
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Laws of software evolution revisited
Lehman, M. M. (1996) · Lecture Notes in Computer Science · doi:10.1007/bfb0017737 · Cited by 572
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
Agile software development: it's about feedback and change
Williams, L., & Cockburn, A. (2003) · Computer · doi:10.1109/mc.2003.1204373 · Cited by 428
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390
A survey of the use of crowdsourcing in software engineering
Mao, K., Capra, L., Harman, M., & Jia, Y. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.09.015 · Cited by 385
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0943 · Cited by 384
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends in Programming Languages · doi:10.1561/2500000010 · Cited by 319
Automated Unit Test Improvement using Large Language Models at Meta
Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B. U., Harman, M., Harper, I., Marginean, A., Sengupta, S., & Wang, E. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663839 · Cited by 316
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
GenderMag: A Method for Evaluating Software's Gender Inclusiveness
Burnett, M., Stumpf, S., Macbeth, J., Makri, S., Beckwith, L., Kwan, I., Peters, A., & Jernigan, W. (2016) · Interacting with Computers · doi:10.1093/iwc/iwv046 · Cited by 293
Generative AI for Software Practitioners
Ebert, C., & Louridas, P. (2023) · IEEE Software · doi:10.1109/ms.2023.3265877 · Cited by 293
Using Large Language Models to Generate JUnit Tests: An Empirical Study
Siddiq, M. L., Santos, J. C. S., Tanvir, R. H., Ulfat, N., Rifat, F. A., & Lopes, V. C. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661216 · Cited by 250
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
How developers search for code: a case study
Sadowski, C., Stolee, K. T., & Elbaum, S. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786855 · Cited by 238
INCOSE International Symposium
(2018) · INCOSE International Symposium · doi:10.1002/(issn)2334-5837 · Cited by 237
Less training, more repairing please: revisiting automated program repair via zero-shot learning
Xia, C. S., & Zhang, L. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549101 · Cited by 231
Shamshiri, S., Just, R., Rojas, J. M., Fraser, G., McMinn, P., & Arcuri, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.86 · Cited by 230
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 230
Burnett, M., Cook, C., & Rothermel, G. (2004) · Communications of the ACM · doi:10.1145/1015864.1015889 · Cited by 215
The Requirements Engineering Handbook
Young, R. R. (2003) · Cited by 208
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
InferFix: End-to-End Program Repair with LLMs
Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., & Svyatkovskiy, A. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613892 · Cited by 191
The (R) Evolution of social media in software engineering
Storey, M.-A., Singer, L., Cleary, B., Figueira Filho, F., & Zagalsky, A. (2014) · Future of Software Engineering Proceedings · doi:10.1145/2593882.2593887 · Cited by 188
Fine-tuning and prompt engineering for large language models-based code review automation
Pornprasit, C., & Tantithamthavorn, C. K. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107523 · Cited by 186
Advancing Requirements Engineering Through Generative AI: Assessing the Role of LLMs
Arora, C., Grundy, J., & Abdelrazek, M. (2024) · Generative AI for Effective Software Development · doi:10.1007/978-3-031-55642-5_6 · Cited by 182
Using ChatGPT in Software Requirements Engineering: A Comprehensive Review
Marques, N., Silva, R. R., & Bernardino, J. (2024) · Future Internet · doi:10.3390/fi16060180 · Cited by 181
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639219 · Cited by 162
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157
Geng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z., Mao, X., & Liao, X. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608134 · Cited by 155
Requirements Classification with Interpretable Machine Learning and Dependency Parsing
Dalpiaz, F., Dell’Anna, D., Aydemir, F. B., & Çevikol, S. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00025 · Cited by 139
Search-based inference of polynomial metamorphic relations
Zhang, J. M., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642994 · Cited by 137
Measuring GitHub Copilot's Impact on Productivity
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2024) · Communications of the ACM · doi:10.1145/3633453 · Cited by 124
Solving the Search for Source Code
Stolee, K. T., Elbaum, S., & Dobos, D. (2014) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2581377 · Cited by 122
A Systematic Review of API Evolution Literature
Lamothe, M., Guéhéneuc, Y.-G., & Shang, W.-Y. (2021) · ACM Computing Surveys · doi:10.1145/3470133 · Cited by 117
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
TOGLL: Correct and Strong Test Oracle Generation with LLMS
Hossain, S. B., & Dwyer, M. B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00098 · Cited by 110
How Well Do Search Engines Support Code Retrieval on the Web?
Sim, S. E., Umarji, M., Ratanotayanon, S., & Lopes, C. V. (2011) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2063239.2063243 · Cited by 102
Pynguin: automated unit test generation for Python
Lukasczyk, S., & Fraser, G. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3516829 · Cited by 101
Test oracle assessment and improvement
Jahangirova, G., Clark, D., Harman, M., & Tonella, P. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931062 · Cited by 88
Bi, Z., Wan, Y., Wang, Z., Zhang, H., Guan, B., Lu, F., Zhang, Z., Sui, Y., Jin, H., & Shi, X. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.138 · Cited by 88
A critical review on the evaluation of automated program repair systems
Liu, K., Li, L., Koyuncu, A., Kim, D., Liu, Z., Klein, J., & Bissyandé, T. F. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110817 · Cited by 79
An Empirical Study on API Usages
Zhong, H., & Mei, H. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2782280 · Cited by 75
Testing Web Enabled Simulation at Scale Using Metamorphic Testing
Ahlgren, J., Berezin, M. E., Bojarczuk, K., Dulskyte, E., Dvortsova, I., George, J., Gucevska, N., Harman, M., Lomelí, M., Meijer, E., et al. (2021) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00023 · Cited by 75
Adaptive Test Generation Using a Large Language ModelPreprint
Schäfer, M., Nadi, S., Eghbali, A., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2302.06527 · Cited by 74
MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation
Blasi, A., Gorla, A., Ernst, M. D., Pezzè, M., & Carzaniga, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111041 · Cited by 69
Lops, A., Narducci, F., Ragone, A., Trizio, M., & Bartolini, C. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962454 · Cited by 66
Malgaonkar, S., Licorish, S. A., & Savarimuthu, B. T. R. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106798 · Cited by 61
Ayerdi, J., Terragni, V., Arrieta, A., Tonella, P., Sagardui, G., & Arratibel, M. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473920 · Cited by 60
Self-citationNo More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
Yuan, Z., Lou, Y., Liu, M., Ding, S., Wang, K., Chen, Y., & Peng, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.04207 · Cited by 56
Bhatia, S., Gandhi, T., Kumar, D., & Jalote, P. (2024) · Proceedings of the 1st International Workshop on Large Language Models for Code · doi:10.1145/3643795.3648396 · Cited by 55
EvoSpex: An Evolutionary Algorithm for Learning Postconditions
Molina, F., Ponzio, P., Aguirre, N., & Frias, M. F. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00112 · Cited by 53
When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention
Guo, L., Wang, Y., Shi, E., Zhong, W., Zhang, H., Chen, J., Zhang, R., Ma, Y., & Zheng, Z. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680343 · Cited by 53
CSNIPPEX: automated synthesis of compilable code snippets from Q&A sites
Terragni, V., Liu, Y., & Cheung, S.-C. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931058 · Cited by 52
Self-citationAccelerating Software Development Using Generative AI: ChatGPT Case Study
Rajbhoj, A., Somase, A., Kulkarni, P., & Kulkarni, V. (2024) · Proceedings of the 17th Innovations in Software Engineering Conference · doi:10.1145/3641399.3641403 · Cited by 50
Programming Languages: Principles and Paradigms
Gabbrielli, M., & Martini, S. (2023) · Undergraduate topics in computer science · doi:10.1007/978-3-031-34144-1 · Cited by 50
Large Language Model-Based Agents for Software Engineering: A Survey
Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2024) · arXiv · doi:10.48550/arxiv.2409.02977 · Cited by 49
Nashaat, M., & Miller, J. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428324 · Cited by 47
Exploring Automated Assertion Generation via Large Language Models
Zhang, Q., Sun, W., Fang, C., Yu, B., Li, H., Yan, M. L., Zhou, J., & Chen, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3699598 · Cited by 47
Understanding Breaking Changes in the Wild
Jayasuriya, D., Terragni, V., Dietrich, J. B., Ou, S., & Blincoe, K. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598147 · Cited by 47
Self-citationGenerating UML Class Diagram using NLP Techniques and Heuristic Rules
Abdelnabi, E. A., Maatuk, A. M., Abdelaziz, T. M., & Elakeili, S. (2020) · International Conference on Sciences and Techniques of Automatic Control and Computer Engineering · doi:10.1109/sta50679.2020.9329301 · Cited by 46
Do Android app users care about accessibility?: an analysis of user reviews on the Google play store
Eler, M. M., Orlandin, L., & Oliveira, A. D. A. (2019) · Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems · doi:10.1145/3357155.3358477 · Cited by 45
Generative AI for Effective Software Development
Nguyen-Duc, A., Abrahamsson, P., & Khomh, F. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5 · Cited by 45
LLMs and Stack Overflow discussions: Reliability, impact, and challenges
Silva, L. D., Samhi, J., & Khomh, F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112541 · Cited by 45
Automatic Discovery and Cleansing of Numerical Metamorphic Relations
Zhang, B., Zhang, H., Chen, J., Hao, D., & Moscato, P. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00035 · Cited by 42
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42
GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming
Ayerdi, J., Terragni, V., Jahangirova, G., Arrieta, A., & Tonella, P. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3407840 · Cited by 39
Self-citationOpenness and requirements: Opportunities and tradeoffs in software ecosystems
Knauss, E., Damian, D., Knauss, A., & Borici, A. (2014) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2014.6912263 · Cited by 36
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation
Luo, Q., Ye, Y., Liang, S., Zhang, Z., Qin, Y., Lu, Y., Wu, Y., Cong, X., Lin, Y., Zhang, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2024.emnlp-demo.46 · Cited by 36
Generative Software EngineeringPreprint
Huang, Y., Chen, Y., Chen, X., Chen, J., Peng, R., Tang, Z., Huang, J., Xu, F., & Zheng, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.02583 · Cited by 36
LLMs and Prompting for Unit Test Generation: A Large-Scale Evaluation
Ouédraogo, W. C., Kaboré, A. K., Tian, H., Song, Y., Koyuncu, A., Klein, J., Lo, D., & Bissyandé, T. F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695330 · Cited by 33
An Empirical Validation of Oracle Improvement
Jahangirova, G., Clark, D., Harman, M., & Tonella, P. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2934409 · Cited by 29
Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning
Juneja, G., Dutta, S., Chakrabarti, S., Manchanda, S., & Chakraborty, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.225 · Cited by 27
Software Testing with Large Language Models: Survey, Landscape, and VisionPreprint
Wang, J., Huang, Y., Chen, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.07221 · Cited by 26
An extended study of syntactic breaking changes in the wild
Jayasuriya, D., Ou, S., Hegde, S., Terragni, V., Dietrich, J. B., & Blincoe, K. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10563-4 · Cited by 22
Self-citationTizard, J., Devine, P., Wang, H., & Blincoe, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3219458 · Cited by 21
Self-citationHigh-level programming languages
Barnett, G. O., & Greenes, R. A. (1970) · Computers and Biomedical Research · doi:10.1016/0010-4809(70)90010-8 · Cited by 20
Software reuse through methodical component reuse and amethodical snippet remixing
Philip, K., Umarji, M., Agarwala, M., Sim, S. E., Gallardo-Valencia, R. E., Lopes, C. V., & Ratanotayanon, S. (2012) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2145204.2145407 · Cited by 20
New visions on metamorphic testing after a quarter of a century of inception
Chen, T. Y., & Tse, T. H. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473136 · Cited by 20
SBFT Tool Competition 2023 - Java Test Case Generation Track
Jahangirova, G., & Terragni, V. (2023) · doi:10.1109/sbft59156.2023.00025 · Cited by 19
Self-citationThe Copyleft Movement: creative Commons Licensing
Broussard, S. L. (2007) · Communication research trends · Cited by 15
MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases
Xu, C., Terragni, V., Zhu, H., Wu, J., & Cheung, S.-C. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3656340 · Cited by 15
Self-citationSrinivas, D., Das, R., Tizpaz-Niari, S., Trivedi, A., & Pacheco, M. L. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.23 · Cited by 12
MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
Xu, C., Chen, S., Wu, J., Cheung, S.-C., Terragni, V., Zhu, H., & Cao, J. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3696020 · Cited by 9
Self-citation"You still have to study" -- On the Security of LLM generated codePreprint
Goetz, S. M., & Schaad, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.07106 · Cited by 7
Together or Apart? Investigating a mediator bot to aggregate bot’s comments on pull requests
Ribeiro, E., Nascimento, R., Steinmacher, I., Xavier, L., Gerosa, M. A., de Paula, H. B., & Wessel, M. S. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00054 · Cited by 4
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation GenerationPreprint
Luo, Q., Ye, Y., Liang, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16667 · Cited by 2
Research works printed but not linked: 16
- Vivek Basanagoudar and Abhijay Srekanth. 2023. Copyright conundrums in generative AI: Github Copilot’s not-so-fair use of open-source licensed code. Journal of Intellectual Property Studies 7 (2023), 58–68.
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https://arxiv.org/abs/2107.03374 · arXiv
- T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology.
- Wenpin Hou and Zhicheng Ji. 2024. A systematic evaluation of large language models for generating programming code. arXiv:2403.00894. Retrieved from https://arxiv.org/abs/2403.00894 · arXiv
- Dong Huang Qingwen Bu Jie Zhang Xiaofei Xie Junjie Chen and Heming Cui. 2023. Bias assessment and mitigation in LLM-based code generation. arXiv:2309.14345. Retrieved from https://arxiv.org/abs/2309.14345 · arXiv
- Dhanushka Jayasuriya, Valerio Terragni, Jens Dietrich, and Kelly Blincoe. 2024. Understanding the impact of APIs behavioral breaking changes on client applications. In Proceedings of the 1st ACM on Software Engineering (PACMSE), 1238–1261.
- Michael Konstantinou Renzo Degiovanni and Mike Papadakis. 2024. Do LLMs generate test oracles that capture the actual or the expected program behaviour? arXiv:2410.21136. Retrieved from https://arxiv.org/abs/2410.21136 · arXiv
- Sebastian Lubos, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. Leveraging LLMs for the quality assurance of software requirements. In Proceedings of the 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 389–397.
- Quang-Hung Luu Huai Liu and Tsong Yueh Chen. 2023. Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing. arXiv:2310.19204. Retrieved from https://arxiv.org/abs/2310.19204 · arXiv
- Garima Malik Mucahit Cevik Devang Parikh and Ayse Basar. 2024. Supervised semantic similarity-based conflict detection algorithm: S3CDA. arXiv:2206.13690. Retrieved from https://arxiv.org/abs/2206.13690 · arXiv
- Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. 2023. From words to watts: Benchmarking the energy costs of large language model inference. In Proceedings of the 2023 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1–9.
- Seung Yeob Shin Fabrizio Pastore Domenico Bianculli and Alexandra Baicoianu. 2024. Towards generating executable metamorphic relations using large language models. arXiv:2401.17019. Retrieved from https://arxiv.org/abs/2401.17019 · arXiv
- Christos Tsigkanos, Pooja Rani, Sebastian Müller, and Timo Kehrer. 2023. Large language models: The next frontier for variable discovery within metamorphic testing? In Proceedings of the 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 678–682.
- Simon Van Oordt and Emitza Guzman. 2021. On the role of user feedback in software evolution: A practitioners’ perspective. In Proceedings of the 2021 IEEE 29th International Requirements Engineering Conference (RE). IEEE, 221–232.
- Zhuokui Xie Yinghao Chen Chen Zhi Shuiguang Deng and Jianwei Yin. 2023. ChatUniTest: A ChatGPT-based automated unit test generation tool. arXiv:2305.04764. Retrieved from https://arxiv.org/abs/2305.04764 · arXiv
- Burak Yetiştiren Işık Özsoy Miray Ayerdem and Eray Tüzün. 2023. Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot Amazon CodeWhisperer and ChatGPT. arXiv:2304.10778. Retrieved from https://arxiv.org/abs/2304.10778 · arXiv
Cited by
Quantum Software Engineering: Roadmap and Challenges Ahead
Murillo, J. M., García-Alonso, J. M., Moguel, E., Barzen, J., Leymann, F., Ali, S., Yue, T., Arcaini, P., Pérez‐Castillo, R., de Guzmán, I. G., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712002 · Cited by 199
Generative AI in Agile Software Development: A Comprehensive Survey
Malladi, N. V., & Reddy, K. S. (2025) · International Conference on Innovative Data Communication Technologies and Application (ICIDCA) · doi:10.1109/icidca66325.2025.11280486 · Cited by 21
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13
Ibáñez, M. B., Estrada, M. L. B., & Cabada, R. Z. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70080 · Cited by 19
Amalfitano, D., Metzger, A., Autili, M., Fulcini, T., Hey, T., Keim, J., Pelliccione, P., Scotti, V., Koziolek, A., Mirandola, R., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3788879 · Cited by 30
Metadata
- Authors
- Terragni, V., Vella, A., Roop, P., & Blincoe, K.
- Affiliations
- University of Auckland (NZ)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-05-26
- Volume
- 34
- Issue
- 5
- Pages
- 1-20
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4406798194
- Cited by
- 62
- References
- 97
- Citations by year
- 2026: 33 · 2025: 29
- FWCI
- 65.00
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Marsden Fund Council
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- large language models, human-AI collaboration, software productivity, future of software development, research challenges
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
