Cited by 125Software Engineering ResearchSoftware Engineering Techniques and PracticesAdvanced Software Engineering Methodologies

The DOI matches its Crossref record; 235 of the 235 references it cites were found in a registry or an index.

Generative Artificial Intelligence for Software Engineering—A Research Agenda

Open access

Nguyen‐Duc, Anh, Cabrero‐Daniel, Beatriz, Przybylek, Adam, Arora, Chetan, Khanna, Dron, Herda, Tomas, Rafiq, Usman, Melegati, Jorge, Guerra, Eduardo, Kemell, Kai‐Kristian, Saari, Mika, Zhang, Zheying, Le, Huy, Quan, Tho, Abrahamsson, Pekka·2025·Software: Practice and Experience

DOI 10.1002/spe.70005

Open this paper in CoChat — free account

Abstract

Context: Generative artificial intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI's ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective: Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method: We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for software engineering. Results: We identified 78 open research questions (RQs) in 11 areas of software engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision‐making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry‐level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions: GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research.

Integrity signals

Registry

Crossref · 10.1002/spe.70005

Cites

235 of 235 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“LLM capabilities have been associated with their size [263], resulting in the rapid growth of model size [264].”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“These frameworks should be designed with input from diverse stakeholders, including ethicists, technologists, and affected communities, to ensure that they address a broad range of ethical concerns [269].”
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Guidelines for snowballing in systematic literature studies and a replication in software engineering
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451
Untitled work
(2011) · Cited by 3,330
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
A survey on semi-supervised learning
van Engelen, J. E., & Hoos, H. H. (2020) · Machine Learning · doi:10.1007/s10994-019-05855-6 · Cited by 2,666
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Artificial Intelligence in Human Resources Management: Challenges and a Path Forward
Tambe, P., Cappelli, P., & Yakubovich, V. (2019) · California Management Review · doi:10.1177/0008125619867910 · Cited by 1,609
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540
Software Engineering for Machine Learning: A Case Study
Amershi, S., Begel, A., Bird, C., DeLine, R. A., Gall, H. C., Kamar, E. S., Nagappan, N., Nushi, B., & Zimmermann, T. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00042 · Cited by 1,401
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
Roles of Artificial Intelligence in Construction Engineering and Management: A Critical Review and Future Trends
Pan, Y., & Zhang, L. (2021) · Automation in Construction · doi:10.1016/j.autcon.2020.103517 · Cited by 1,336
Evidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232
Model-driven Development of Complex Software: A Research Roadmap
France, R., & Rumpe⋆, B. (2007) · Future of Software Engineering (FOSE) · doi:10.1109/fose.2007.14 · Cited by 1,133
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Artificial intelligence in the construction industry: a review of present status, opportunities and future challenges
Abioye, S. O., Oyedele, L. O., Akanbi, L., Ajayi, A., Davila Delgado, J. M., Bilal, M., Akinade, O. O., & Ahmed, A. (2021) · Journal of Building Engineering · doi:10.1016/j.jobe.2021.103299 · Cited by 1,015
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
(2021) · doi:10.18653/v1/2021.acl-long · Cited by 926
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · cited 2×
“On top of that, there is a gender gap in technology-related study fields [200, 202] and a soft skills gap [197, 203] that make it even harder to find professionals and have balanced teams able to detect and correct unfair biases, for example, caused by the selection of data sources [204].”
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.8 · Cited by 810
SERVICE-ORIENTED COMPUTING: A RESEARCH ROADMAP
Papazoglou, M., Traverso, P., Dustdar, S., & Leymann, F. (2008) · International Journal of Cooperative Information Systems · doi:10.1142/s0218843008001816 · Cited by 773
Expediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752
The effect of generative artificial intelligence (AI)-based tool use on students' computational thinking skills, programming self-efficacy and motivation
Yilmaz, R., & Karaoglan Yilmaz, F. G. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100147 · Cited by 745
Continuous software engineering: A roadmap and agenda
Fitzgerald, B. F., & Stol, K.-J. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 742
Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education
Qadir, J. (2023) · 2023 IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon54358.2023.10125121 · Cited by 707 · cited 1×
“Current AI approaches focus on finding questions of similar difficulty to a given question [215].”
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
The GenAI is out of the bottle: generative artificial intelligence from a business model innovation perspective
Kanbach, D. K., Heiduk, L., Blueher, G., Schreiter, M., & Lahmann, A. D. F. (2023) · Review of Managerial Science · doi:10.1007/s11846-023-00696-z · Cited by 634
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. (2022) · neural information processing systems · doi:10.52202/068431-0142 · Cited by 634
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598
The Four Steps to the Epiphany: Successful Strategies for Products that Win
Blank, S. (2013) · Cited by 581
Large Language Models in Machine Translation
Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. B. (2007) · Cited by 574
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557
A systematic literature review on agile requirements engineering practices and challenges
Inayat, I., Salim, S. S., Marczak, S. D. S., Daneva, M., & Shamshirband, S. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.10.046 · Cited by 531
Explainable AI Methods - A Brief Overview
Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04083-2_2 · Cited by 518
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
GraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
The Focus Group Research Handbook
Edmunds, H. (1999) · The Bottom Line Managing Library Finances · doi:10.1108/bl.1999.12.3.46.1 · Cited by 467
A sense of community: A research agenda for software ecosystems
Jansen, S., Finkelstein, A., & Brinkkemper, S. (2009) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2009.5070978 · Cited by 467
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
Self-citation
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A., Bui, N., Li, J., & Hoi, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.68 · Cited by 459
System Engineering Management
Blanchard, B. S., & Blyler, J. E. (2016) · Wiley eBooks · doi:10.1002/9781119178798 · Cited by 441
The Impact of Artificial Intelligence on Workers' Skills: Upskilling and Reskilling in Organisations
Morandini, S., Fraboni, F., De Angelis, M., Puzzo, G., Giusino, D., & Pietrantoni, L. (2023) · Informing Science: The International Journal of an Emerging Transdiscipline · doi:10.28945/5078 · Cited by 439 · cited 1×
“Moradini conducted a narrative review to analyze research and practice on the impact of AI on human skills in organizations [235].”
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
Agile requirements engineering practices and challenges: an empirical study
Ramesh, B., Cao, L., & Baskerville, R. (2007) · Information Systems Journal · doi:10.1111/j.1365-2575.2007.00259.x · Cited by 428
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 426
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Fairness, Accountability, Transparency, and Ethics (FATE) in Artificial Intelligence (AI) and higher education: A systematic review
Memarian, B., & Doleck, T. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100152 · Cited by 416
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393
Generative Artificial Intelligence: Trends and Prospects
Jovanovic, M., & Campbell, M. (2022) · Computer · doi:10.1109/mc.2022.3192720 · Cited by 357
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3657432 · Cited by 356
Large language models and the perils of their hallucinations
Azamfirei, R., Kudchadkar, S. R., & Fackler, J. C. (2023) · Critical Care · doi:10.1186/s13054-023-04393-x · Cited by 352
Cost of Experimentation and the Evolution of Venture Capital
Ewens, M., Nanda, R., & Rhodes‐Kropf, M. (2015) · Journal of Financial Economics · doi:10.1016/j.jfineco.2018.03.001 · Cited by 351
ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design
White, J., Hays, S., Fu, Q., Spencer-Smith, J. B., & Schmidt, D. C. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_4 · Cited by 344
Automated Assessment in Computer Science Education: A State-of-the-Art Review
Paiva, J. C., Leal, J. P., & Figueira, Á. (2022) · ACM Transactions on Computing Education · doi:10.1145/3513140 · Cited by 342
Measuring Program Comprehension: A Large-Scale Field Study with Professionals
Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A. E., & Li, S. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2734091 · Cited by 326
From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and FuturePreprint
Jin, H., Huang, L., Cai, H., Yan, J., Li, B., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.02479 · Cited by 305
Large Language Model-Based Agents for Software Engineering: A Survey
Jun-wei, L., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3796507 · Cited by 299
Generative AI for Software Practitioners
Ebert, C., & Louridas, P. (2023) · IEEE Software · doi:10.1109/ms.2023.3265877 · Cited by 293
THE STRENGTH OF WEAK TIES YOU CAN TRUST: THE MEDIATING ROLE OF TRUST IN EFFECTIVE KNOWLEDGE TRANSFER.
Levin, D. Z., Cross, R., & Abrams, L. C. (2002) · Academy of Management Proceedings · doi:10.5465/apbpp.2002.7517527 · Cited by 291
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
ChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
Artificial intelligence in product lifecycle management
Wang, L., Liu, Z., Liu, A., & Tao, F. (2021) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-021-06882-1 · Cited by 267
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Self-collaboration code generation via ChatGPT
Dong, Y., Jiang, X., Jin, Z., & Li, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672459 · Cited by 256
The AI Digital Revolution in Innovation: A Conceptual Framework of Artificial Intelligence Technologies for the Management of Innovation
Brem, A., Giones, F., & Werle, M. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3109983 · Cited by 249
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test GenerationPreprint
Yuan, Z.-Q., Lou, Y., Liu, M.-W., Ding, S., Wang, K.-X., Chen, Y.-X., & Peng, X. (2023) · arXiv.org · doi:10.48550/arxiv.2305.04207 · Cited by 239
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235
Do Code and Comments Co-Evolve? On the Relation between Source Code and Comment Changes
Fluri, B., Würsch, M., & Gall, H. C. (2007) · Proceedings - Working Conference on Reverse Engineering · doi:10.1109/wcre.2007.21 · Cited by 231
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226
Gorilla: Large Language Model Connected with Massive APIs
Patil, S., Zhang, T., Wang, X., & Gonzalez, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-4020 · Cited by 226
NoRBERT: Transfer Learning for Requirements Classification
Hey, T., Keim, J., Koziolek, A., & Tichy, W. F. (2020) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re48521.2020.00028 · Cited by 216
A research agenda for distributed software development
Sengupta, B., Chandra, S., & Sinha, V. S. (2006) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1134285.1134402 · Cited by 213
Improved Trust in Human-Robot Collaboration With ChatGPT
Ye, Y., You, H., & Du, J. (2023) · IEEE Access · doi:10.1109/access.2023.3282111 · Cited by 213
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 212
Using the focus group method in software engineering: obtaining practitioner and user experiences
Kontio, J., Lehtola, L., & Bragge, J. (2004) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2004.1334914 · Cited by 211
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H. A., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179420 · Cited by 206
Essentials of project and systems engineering management
(1997) · Choice Reviews Online · doi:10.5860/choice.34-4492 · Cited by 206
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning
Geng, M., Wang, S., Dong, D., Wang, H., Li, G., Jin, Z. G., Mao, X., & Liao, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3608134 · Cited by 199
On the Performance of Hybrid Search Strategies for Systematic Literature Reviews in Software Engineering
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., & Wohlin, C. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106294 · Cited by 191
Advancing Requirements Engineering Through Generative AI: Assessing the Role of LLMs
Arora, C., Grundy, J., & Abdelrazek, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_6 · Cited by 188
Self-citation
Research Agenda in Cloud TechnologiesPreprint
Sriram, I., & Khajeh‐Hosseini, A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1001.3259 · Cited by 188
How do professional developers comprehend software?
Roehm, T., Tiarks, R., Koschke, R., & Maalej, W. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227188 · Cited by 187
The Focus Group Method as an Empirical Tool in Software Engineering
Kontio, J., Bragge, J., & Lehtola, L. (2007) · Springer eBooks · doi:10.1007/978-1-84800-044-5_4 · Cited by 184
The gender gap in higher STEM studies: A systematic literature review
Verdugo-Castro, S., García‐Holgado, A., & Gómez, M. C. S. (2022) · Heliyon · doi:10.1016/j.heliyon.2022.e10300 · Cited by 178
How ChatGPT Will Change Software Engineering Education
Daun, M., & Brings, J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588815 · Cited by 172
A systematic literature review of literature reviews in software testing
Garousi, V., & Mäntylä, M. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.09.002 · Cited by 163
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
Learning Performance-Improving Code Edits
Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A. (2023) · arXiv · doi:10.48550/arxiv.2302.07867 · Cited by 146
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 133
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Self-citation
Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Lyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662036 · Cited by 131
Sustainability in software engineering: a systematic literature review
Penzenstadler, B., Bauer, V., Calero, C., & Franch, X. (2012) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2012.0004 · Cited by 130
Reducing the Carbon Impact of Generative AI Inference (today and in 2035)
Chien, A. A., Lin, L., Nguyen, H., Rao, V., Sharma, T., & Wijayawardana, R. (2023) · Proceedings of the 2nd Workshop on Sustainable Computer Systems · doi:10.1145/3604930.3605705 · Cited by 128
Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?
Asare, O., Nagappan, M., & Asokan, N. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10380-1 · Cited by 128
Evaluating LLM-generated Worked Examples in an Introductory Programming Course
Jury, B., Lorusso, A., Leinonen, J., Denny, P. C., & Luxton-Reilly, A. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636252 · Cited by 125
Generative AI for Requirements Engineering: A Systematic Literature Review
Cheng, H., Husen, J. H., Lu, Y., Racharak, T., Yoshioka, N., Ubayashi, N., & Washizaki, H. (2025) · Software Practice and Experience · doi:10.1002/spe.70029 · Cited by 125
A survey on large language models for software engineering
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., Yang, Y., & Chen, Z. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4670-0 · Cited by 124
Applications of AI in classical software engineering
Barenkamp, M., Rebstadt, J., & Thomas, O. (2020) · AI Perspectives · doi:10.1186/s42467-020-00005-4 · Cited by 123
Generating accurate assert statements for unit test cases using pretrained transformers
Tufano, M., Drain, D., Svyatkovskiy, A., & Sundaresan, N. (2022) · doi:10.1145/3524481.3527220 · Cited by 122
Automatic Code Summarization via ChatGPT: How Far Are We?Preprint
Sun, W.-S., Fang, C., You, Y., Miao, Y., Liu, Y., Li, Y., Deng, G.-L., Huang, S., Chen, Y., Zhang, Q.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.12865 · Cited by 121
SelfEvolve: A Code Evolution Framework via Large Language ModelsPreprint
Jiang, S., Wang, Y., & Wang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02907 · Cited by 120
Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117
Is GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116
An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project
Rasnayaka, S., Wang, G. X., Shariffdeen, R., & Iyer, G. N. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648379 · Cited by 113
Rapid Reviews in Software Engineering
Cartaxo, B., Pinto, G., & Soares, S. (2020) · Springer eBooks · doi:10.1007/978-3-030-32489-6_13 · Cited by 112
Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes
Ronanki, K., Berger, C., & Horkoff, J. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00061 · Cited by 110
Jailbreaking ChatGPT via Prompt Engineering: An Empirical StudyPreprint
Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Wang, K., & Liu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13860 · Cited by 109
Engineering AI Systems: A Research Agenda
Bosch, J., Olsson, H. H., & Crnkovic, I. (2021) · Advances in Systems Analysis, Software Engineering, and High Performance Computing · doi:10.4018/978-1-7998-5101-1.ch001 · Cited by 108
Online self-assessment and students’ success in higher education institutions
Ćukušić, M., Garača, Ž., & Jadrić, M. (2014) · Computers & Education · doi:10.1016/j.compedu.2013.10.018 · Cited by 106
Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B. E., Liffiton, M., Šavelka, J., & Denny, P. C. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 103
The Current State of Industrial Practice in Artificial Intelligence Ethics
Vakkuri, V., Kemell, K., Kultanen, J., & Abrahamsson, P. (2020) · IEEE Software · doi:10.1109/ms.2020.2985621 · Cited by 98
Self-citation
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step
Li, Z., Wang, Z., & Shang, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.49 · Cited by 97
Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs
Ma, W., Liu, S., Liu, S., Hu, Q., Liu, Y., Wang, W., Zhang, C., Nie, L., Li, L., Liu, Y., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3818607 · Cited by 90
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions About Code
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2023) · International Conference on Computer Supported Education · doi:10.5220/0011996900003470 · Cited by 89
The soft skills gap: a bottleneck in the talent supply in emerging economies
Dubey, R. S., Paul, J., & Tewari, V. (2021) · The International Journal of Human Resource Management · doi:10.1080/09585192.2020.1871399 · Cited by 87
Multitask Prompt Tuning Enables Parameter-Efficient Transfer LearningPreprint
Wang, Z., Panda, R., Karlinsky, L., Feris, R., Sun, H., & Kim, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.02861 · Cited by 87
Learning Deep Semantics for Test Completion
Nie, P., Banerjee, R., Li, J. J., Mooney, R. J., & Gligoric, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00178 · Cited by 86
A Pilot Study Integrating an AI-driven Chatbot in an Introductory Programming Course
Verleger, M., & Pembridge, J. (2018) · 2018 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2018.8659282 · Cited by 86
Automated handling of anaphoric ambiguity in requirements
Ezzini, S., Abualhaija, S., Arora, C., & Sabetzadeh, M. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510157 · Cited by 85
Self-citation
Using Transfer Learning for Code-Related Tasks
Mastropaolo, A., Cooper, N., Palacio, D. N., Scalabrino, S., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183297 · Cited by 85
Generative artificial intelligence assistants in software development education: A vision for integrating generative artificial intelligence into educational practice, not instinctively defending against it
Bull, C., & Kharrufa, A. (2024) · IEEE Software · doi:10.1109/ms.2023.3300574 · Cited by 82
A Human Study of Comprehension and Code Summarization
Stapleton, S., Gambhir, Y., LeClair, A., Eberhart, Z., Weimer, W. R., Leach, K., & Huang, Y. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389258 · Cited by 82
Extracting Domain Models from Textual Requirements in the Era of Large Language Models
Arulmohan, S., Meurs, M., & Mosser, S. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1109/models-c59198.2023.00096 · Cited by 82
The Expectations of Project Managers from Artificial Intelligence: A Delphi Study
Holzmann, V., Zitter, D., & Peshkess, S. (2022) · Project Management Journal · doi:10.1177/87569728211061779 · Cited by 80
ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Design
Lanzi, P. L., & Loiacono, D. (2023) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3583131.3590351 · Cited by 80
Toward sustainable software engineering (NIER track)
Amsel, N., Ibrahim, Z., Malik, A., & Tomlinson, B. (2011) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1985793.1985964 · Cited by 79
GPT2SP: A Transformer-Based Agile Story Point Estimation Approach
Fu, M. C., & Tantithamthavorn, C. K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3158252 · Cited by 77
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
Yetiştiren, B., Özsoy, I., Ayerdem, M., & Tüzün, E. (2023) · arXiv · doi:10.48550/arxiv.2304.10778 · Cited by 75
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
Steenhoek, B. J., Tufano, M., Sundaresan, N., & Svyatkovskiy, A. (2025) · IEEE/ACM International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest) · doi:10.1109/deeptest66595.2025.00011 · Cited by 72
Requirements engineering framework for human-centered artificial intelligence software systems
Ahmad, K., Abdelrazek, M., Arora, C., Baniya, A. A., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110455 · Cited by 70
Self-citation
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step
Zhong, L., Wang, Z., & Shang, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16906 · Cited by 69
Assessing the Benefits of ChatGPT for Business: An Empirical Study on Organizational Performance
Chu, M.-N. (2023) · IEEE Access · doi:10.1109/access.2023.3297447 · Cited by 69
Software maintenance seen as a knowledge management issue
Anquetil, N., de Oliveira, K. M., de Sousa, K. D., & Dias, M. G. B. (2006) · Information and Software Technology · doi:10.1016/j.infsof.2006.07.007 · Cited by 69
Optimizing Project Management using Artificial Intelligence
Prifti, V. (2022) · European Journal of Formal Sciences and Engineering · doi:10.26417/667hri67 · Cited by 68
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 67
Comparing Software Developers with ChatGPT: An Empirical InvestigationPreprint
Nascimento, N., Alencar, P., & Cowan, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11837 · Cited by 65
Ethical Aspects of ChatGPT in Software Engineering Research
Akbar, M. A., Khan, A. A., & Liang, P. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3318183 · Cited by 65
Improving learning experiences in software engineering capstone courses using artificial intelligence virtual assistants
González, L. A., Neyem, A., Contreras‐McKay, I., & Molina, D. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22526 · Cited by 63
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 63
Lecturer's Apprentice: A Chatbot for Assisting Novice Programmers
Ismail, M., & Ade-Ibijola, A. (2019) · International Multidisciplinary Information Technology and Engineering Conference (IMITEC) · doi:10.1109/imitec45504.2019.9015857 · Cited by 62
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Tian, Y., Yan, W., Yang, Q., Zhao, X., Chen, Q., Wang, W., Luo, Z., Ma, L., & Song, D. X. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34717 · Cited by 62
On the transferability of pre-trained language models for low-resource programming languages
Chen, F., Fard, F. H., Lo, D., & Bryksin, T. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527917 · Cited by 59
Identifying and Mitigating the Security Risks of Generative AI
Barrett, C., Boyd, B., Bursztein, E., Carlini, N., Chen, B., Choi, J., Chowdhury, A. R., Christodorescu, M., Datta, A., Feizi, S., et al. (2023) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000041 · Cited by 57
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Savelka, J., Agarwal, A., Bogart, C., & Sakr, M. (2023) · arXiv · doi:10.48550/arxiv.2303.08033 · Cited by 56
The Influence of Human Aspects on Requirements Engineering-related Activities: Software Practitioners’ Perspective
Hidellaarachchi, D., Grundy, J., Hoda, R., & Mueller, I. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3546943 · Cited by 55
Empirical Evaluation of ChatGPT on Requirements Information Retrieval Under Zero-Shot Setting
Zhang, J., Chen, Y., Liu, C., Niu, N., & Wang, Y. (2023) · International Conference on Intelligent Computing and Next Generation Networks (ICNGN) · doi:10.1109/icngn59831.2023.10396810 · Cited by 55
Greening Large Language Models of Code
Shi, J., Yang, Z., Kang, H. J., Xu, B., He, J., & Lo, D. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1145/3639475.3640097 · Cited by 54
Machine Learning for Software Engineering: A Tertiary Study
Kotti, Z., Galanopoulou, R., & Spinellis, D. D. (2022) · ACM Computing Surveys · doi:10.1145/3572905 · Cited by 54
CRaDLe: Deep code retrieval based on semantic Dependency Learning
Gu, W., Li, Z., Gao, C., Wang, C., Zhang, H., Xu, Z., & Lyu, M. R.-T. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.04.019 · Cited by 52
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
Handling requirements dependencies in agile projects: A focus group with agile software development practitioners
Martakis, A., & Daneva, M. (2013) · International Conference on Research Challenges in Information Science (RCIS) · doi:10.1109/rcis.2013.6577679 · Cited by 48
Automated Question Answering for Improved Understanding of Compliance Requirements: A Multi-Document Study
Abualhaija, S., Arora, C., Sleimi, A., & Briand, L. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00011 · Cited by 46
Self-citation
Pair Programming Education Aided by ChatGPT
Banić, B., Konecki, M., & Konecki, M. (2023) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro57284.2023.10159727 · Cited by 43
Sustainable Software Engineering: Process and Quality Models, Life Cycle, and Social Aspects
Naumann, S., Kern, E., Dick, M., & Johann, T. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-09228-7_11 · Cited by 42
Domain Adaptation for Code Model-Based Unit Test Case Generation
Shin, J., Hashtroudi, S., Hemmati, H., & Wang, S. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680354 · Cited by 41
Execution-based Code Generation using Deep Reinforcement Learning
Shojaee, P., Jain, A., Tipirneni, S., & Reddy, C. K. (2023) · arXiv · doi:10.48550/arxiv.2301.13816 · Cited by 39
Towards an Early Stage Software Startups Evolution Model
Nguyen‐Duc, A., Shah, S. M. A., & Ambrahamsson, P. (2016) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa.2016.21 · Cited by 38
Self-citation
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. J. F. (2023) · arXiv · doi:10.48550/arxiv.2303.08896 · Cited by 38
Adopting Collaborative Games into Agile Requirements Engineering
Przybyłek, A., & Zakrzewski, M. (2018) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0006681900540064 · Cited by 37
Team Performance in Agile Development Teams: Findings from 18 Focus Groups
Dingsøyr, T., & Lindsjørn, Y. (2013) · Lecture notes in business information processing · doi:10.1007/978-3-642-38314-4_4 · Cited by 36
Effective Software Maintenance and Evolution
Jarzabek, S. (2007) · Auerbach Publications eBooks · doi:10.1201/9781420013115 · Cited by 35
Exploring the Potential of Artificial Intelligence Program Generators in Computer Programming Education for Students
Philbin, C. A. (2023) · ACM Inroads · doi:10.1145/3610406 · Cited by 34
AI-assisted coding: Experiments with GPT-4Preprint
Poldrack, R. A., Lu, T., & Beguš, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13187 · Cited by 32
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
(2024) · Proceedings of the 30th Conference on Pattern Languages of Programs · doi:10.64346/plop2023p05 · Cited by 32 · cited 2×
“To address the RQs in this domain, it is imperative to comprehend the roles GenAI tools can assume in the software design process and the approaches for engaging with them using prompt patterns, as discussed in [86].”
An Empirical Study on Code Comment CompletionPreprint
Mastropaolo, A., Aghajani, E., Pascarella, L., & Bavota, G. (2021) · IEEE International Conference on Software Maintenance and Evolution · doi:10.48550/arxiv.2107.10544 · Cited by 31
Empowering Business Transformation: The Positive Impact and Ethical Considerations of Generative AI in Software Product Management – A Systematic Literature Review
Parikh, N. A. (2023) · Advances in Business Strategy and Competitive Advantage · doi:10.4018/979-8-3693-1634-4.ch016 · Cited by 29
Pyo, a Chatbot Assistant for Introductory Programming Students
Carreira, G., Silva, L., Mendes, A. J., & Oliveira, H. G. (2022) · International Symposium on Computers in Education · doi:10.1109/siie56031.2022.9982349 · Cited by 27
Program Risk Contingency Budget Planning
Khamooshi, H., & Cioffi, D. F. (2009) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2008.927818 · Cited by 27
Engineering Project Management
Parker, R. (1994) · R and D Management · doi:10.1111/j.1467-9310.1994.tb00851.x · Cited by 26
Generating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 26
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23

Research works printed but not linked: 52

S.Ouyang J. M.Zhang M.Harman andM.Wang “LLM is Like a Box of Chocolates: The Non‐Determinism of ChatGPT in Code Generation ”2023 arXiv:2308.02828 [cs] https://doi.org/10.48550/arXiv.2308.02828. · DOI · Paper page
10.48550/arXiv.2409.06741 · DOI
Brown T. B., 2020, Proceedings of the 34th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2020‐12‐06) (NIPS'20), 1877
A.RadfordandK.Narasimhan “Improving Language Understanding by Generative Pre‐Training ”2018 https://www.semanticscholar.org/paper/Improving‐Language‐Understanding‐by‐Generative‐Radford‐Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035.
Heafield K., 2013, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 690
Radford A., 2019, Language Models Are Unsupervised Multitask Learners, OpenAI Blog, 1, 9
Q.Zhang M.Chen A.Bukharin et al. “Adaptive Budget Allocation for Parameter‐Efficient Fine‐Tuning ”2023 arXiv:2303.10512 [cs.CL]. · arXiv
J.Zhang Y.Chen N.Niu andC.Liu “A Preliminary Evaluation of ChatGPT in Requirements Information Retrieval ”2023 arXiv Preprint arXiv:2304.12562. · DOI
Ezzini S., 2023, Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 1277
Luo X., 2023, Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE ‘22)
Chen B., 2023, MoDRE Workshop at Requirement Engineering Conference
Nahar N., 2016, Proceedings of the 2016 5th International Conference on Informatics, Electronics and Vision (ICIEV), 35
Ahmad A., 2023, Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (Oulu, Finland) (EASE ‘23), New York, NY, 279
Maranhão Junior J. J., 2023, AI‐Assisted Agile Software Development Workshop at XP2023 Conference
T.StojanovicandS. D.Lazarević “The Application of ChatGPT for Identification of Microservices ”3 no.1(2023):99–105 Number: 1 https://ebt.rs/journals/index.php/conf‐proc/article/view/181.
Gamma E., 1995, Design Patterns
Viviani L., 2023, International Conference on Agile Software Development, 77
10.1109/ICSE-Companion55297.2022.9793778 · DOI · Paper page
Siddiq M. L., 2023, Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language‐Based Software Engineering (NLBSE) (2023–05), 56
Y.Dong J.Xue Z.Jin andG.Li “Self‐Collaboration Code Generation via ChatGPT ”2023 arXiv:2304.07590 [cs.SE]. · arXiv
Jiao Sun Q., 2022, Proeedings of the 27th International Conference on Intelligent User Interfaces (New York, NY, USA, 2022‐03‐22) (IUI ‘22), 212
H.Li H.Yu Y.Zhai andZ.Qian “The Hitchhiker's Guide to Program Analysis: A Journey With Large Language Models ”2023 arXiv:2308.00245 [cs.SE]. · arXiv
Salza P., 2021, On the Effectiveness of Transfer Learning for Code Search, 49, no. 4, 1804
Wei M., 2022, Proceedings of the 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022–05), 376
Mastropaolo A., 2021, Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 336
X.Liu Y.Zheng D.Zhengxiao et al. “GPT Understands Too ”(2021) arXiv:2103.10385 [cs.CL]. · arXiv
R.Li L. B.Allal Y.Zi et al. “StarCoder: May the Source be With You!”(2023) arXiv:2305.06161 [cs.CL]. · arXiv
Planning S., 2002, The Economic Impacts of Inadequate Infrastructure for Software Testing, National Institute of Standards and Technology, 1
10.1145/3572905 · DOI
W.Ma S.Liu W.Wang et al. “The Scope of ChatGPT in Software Engineering: A Thorough Investigation ”(2023) arXiv Preprint arXiv:2305.12138. · arXiv
J.White S.Hays F.Quchen J.Spencer‐Smith andD. C.Schmidt “Chatgpt Prompt Patterns for Improving Code Quality Refactoring Requirements Elicitation and Software Design ”(2023) arXiv Preprint arXiv:2303.07839. · DOI
R.Feldt S.Kang J.Yoon andS.Yoo “Towards Autonomous Testing Agents via Conversational Large Language Models ”(2023) arXiv Preprint arXiv:2306.05152. · DOI · Paper page
W.Sun C.Fang Y.You et al. “Automatic Code Summarization via ChatGPT: How Far Are We?”(2023) arXiv:2305.12865 [cs.SE]. · arXiv
Ahmed T., 2023, Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE'22)
Q.Zhang C.Fang W.Sun et al. “Boosting Automated Patch Correctness Prediction via Pre‐Trained Language Model ”(2023) arXiv:2301.12453 [cs.SE]. · arXiv
X.Hou Y.Zhao Y.Liu et al. “Large Language Models for Software Engineering: A Systematic Literature Review ”(2023) arXiv:2308.10620 [cs.SE]. · arXiv
Sommerville I., 2016, Software Engineering
Weisz J. D., 2021, Proceedings of the 26th International Conference on Intelligent User Interfaces (New York, NY, USA, 2021‐04‐14) (IUI ‘21), 402
Steven I., 2023, Proceedings of the 28th International Conference on Intelligent User Interfaces (New York, NY, USA, 2023‐03‐27) (IUI ‘23), 491
Anthony P., 2002, Ammeter and Janet M Dukerich. 2002. Leadership, Team Building, and Team Member Characteristics in High Performance Project Teams, Engineering Management Journal, 14, 3
Dam H. K., 2019, Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE‐NIER), 41
Haytham H., 2020, Elmousalami. 2020. Comparison of Artificial Intelligence Techniques for Project Conceptual Cost Prediction: A Case Study and Comparative Analysis, IEEE Transactions on Engineering Management, 68, 183
OpenAI “GPT‐4 Technical Report ”(2023) arXiv:2303.08774 [cs.CL]. · arXiv
D.Adiwardana M.‐T.Luong D. R.So et al. “Towards a Human‐Like Open‐Domain Chatbot. CoRR Abs/2001.09977 ”(2020) arXiv:2001.09977 https://arxiv.org/abs/2001.09977. · arXiv
European Commission, 2021, EUR‐Lex ‐ 52021PC0206 ‐ EN ‐ EUR‐Lex
Binkis M., 2021, Information and Software Technologies (Cham, 2021) (Communications in Computer and Information Science), 367
Berrezueta‐Guzman J., 2003, Proceedings of the 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) (2023–08), 147
Kazemitabaar M., 2024, Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (Koli Calling ‘23), 1
Balse R., 2023, Proceedings of the 16th Annual ACM India Compute Conference (COMPUTE ‘23), 49
10.48550/arXiv.2304.12269 · DOI
W.Ma S.Liu W.Wang et al. “The Scope of ChatGPT in Software Engineering: A Thorough Investigation ”2023 arXiv:2305.12138 [cs] https://doi.org/10.48550/arXiv.2305.12138. · DOI
F.Liu L.Yang S.Lin et al. “Exploring and Evaluating Hallucinations in LLM‐Powered Code Generation ”2024 arXiv:2404.00971 [cs] https://doi.org/10.48550/arXiv.2404.00971. · DOI · Paper page

Sorted by citations

LLM-Based Agents for Automating the Enhancement of User Story Quality: An Early Report
Zhang, Z., Rayhan, M., Herda, T., Goisauf, M., & Abrahamsson, P. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61154-4_8 · Cited by 67
Self-citation
Large language models for qualitative research in software engineering: exploring opportunities and challenges
Bano, M., Hoda, R., Zowghi, D., & Treude, C. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00407-8 · Cited by 63
Generative AI Meets Knowledge Management: Insights From Software Development Practices
Kirchner, K., Bolisani, E., Kassaneh, T. C., Scarso, E., & Taraghi, N. (2025) · Knowledge and Process Management · doi:10.1002/kpm.70004 · Cited by 40
Democratizing Software Engineering through Generative AI and Vibe Coding: The Evolution of No-Code Development
Gadde, A. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.4.66 · Cited by 31
A Review on Vibe Coding: Fundamentals, State-of-the-art, Challenges and Future DirectionsPreprint
Ray, P. P. (2025) · doi:10.36227/techrxiv.174681482.27435614/v1 · Cited by 27

See all 125

Authors: Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., Saari, M., Zhang, Z., Le, H., Quan, T., & Abrahamsson, P.
Affiliations: Norwegian University of Science and Technology (NO)
University of Gothenburg (SE)
Gdansk University of Technology Gdansk Poland
Monash University (AU)
Free University of Bozen-Bolzano (IT)
Austrian Post—Konzern IT Vienna Austria
University of Helsinki (FI)
Tampere University (FI)
Vietnam National University Ho Chi Minh City (VN)
Corresponding author: Nguyen‐Duc, Anh
ORCID: Nguyen‐Duc, Anh
Cabrero‐Daniel, Beatriz
Arora, Chetan
Khanna, Dron
Herda, Tomas
Rafiq, Usman
Melegati, Jorge
Guerra, Eduardo
Kemell, Kai‐Kristian
Saari, Mika
Zhang, Zheying
Quan, Tho
Abrahamsson, Pekka
Venue: Software: Practice and Experience
Publisher: Wiley
Published: 2025-07-29
Volume: 55
Issue: 11
Pages: 1806-1843
ISSN: 0038-0644 (ISSN-L)
1097-024X
DOI: 10.1002/spe.70005
OpenAlex: W4412899734
Cited by: 60
References: 222
Citations by year: 2026: 47 · 2025: 13
FWCI: 62.39
Type: journal article
Language: en
Open access: green
hdl.handle.net
Licence: other-oa
onlinelibrary.wiley.com/termsAndConditions#vor (vor)
Locations: Software Practice and Experience (publishedVersion)
Työväentutkimus Vuosikirja (submittedVersion, other-oa)
Chalmers Research (Chalmers University of Technology) (submittedVersion)
Open Research (University of Surrey) (submittedVersion, cc-by)
Topics: Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords: generative AI, software engineering, research agenda, open research questions, ChatGPT, GitHub Copilot, software development automation, decision support, requirements engineering, software design, software engineering education, software maintenance, quality assurance, dependability, transparency, sustainability, data accessibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Generative Artificial Intelligence for Software Engineering—A Research Agenda

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Citations

Cites

Cited by

Related

Metadata

Record

About these checks