Cited by 125Software Engineering ResearchSoftware Engineering Techniques and PracticesAdvanced Software Engineering Methodologies

The DOI matches its Crossref record; 235 of the 235 references it cites were found in a registry or an index.

Generative Artificial Intelligence for Software Engineering—A Research Agenda

Open access

Abstract

Context: Generative artificial intelligence (GenAI) tools have become increasingly prevalent in software development, offering assistance to various managerial and technical project activities. Notable examples of these tools include OpenAI's ChatGPT, GitHub Copilot, and Amazon CodeWhisperer. Objective: Although many recent publications have explored and evaluated the application of GenAI, a comprehensive understanding of the current development, applications, limitations, and open challenges remains unclear to many. Particularly, we do not have an overall picture of the current state of GenAI technology in practical software engineering usage scenarios. Method: We conducted a literature review and focus groups for a duration of five months to develop a research agenda on GenAI for software engineering. Results: We identified 78 open research questions (RQs) in 11 areas of software engineering. Our results show that it is possible to explore the adoption of GenAI in partial automation and support decision‐making in all software development activities. While the current literature is skewed toward software implementation, quality assurance and software maintenance, other areas, such as requirements engineering, software design, and software engineering education, would need further research attention. Common considerations when implementing GenAI include industry‐level assessment, dependability and accuracy, data accessibility, transparency, and sustainability aspects associated with the technology. Conclusions: GenAI is bringing significant changes to the field of software engineering. Nevertheless, the state of research on the topic still remains immature. We believe that this research agenda holds significance and practical value for informing both researchers and practitioners about current applications and guiding future research.

Overview

Integrity signals

Registry

Crossref · 10.1002/spe.70005

Cites

235 of 235 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 222 references

Research works printed but not linked: 52

  1. S.Ouyang J. M.Zhang M.Harman andM.Wang “LLM is Like a Box of Chocolates: The Non‐Determinism of ChatGPT in Code Generation ”2023 arXiv:2308.02828 [cs] https://doi.org/10.48550/arXiv.2308.02828. · DOI · Paper page
  2. 10.48550/arXiv.2409.06741 · DOI
  3. Brown T. B., 2020, Proceedings of the 34th International Conference on Neural Information Processing Systems (Red Hook, NY, USA, 2020‐12‐06) (NIPS'20), 1877
  4. A.RadfordandK.Narasimhan “Improving Language Understanding by Generative Pre‐Training ”2018 https://www.semanticscholar.org/paper/Improving‐Language‐Understanding‐by‐Generative‐Radford‐Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035.
  5. Heafield K., 2013, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 690
  6. Radford A., 2019, Language Models Are Unsupervised Multitask Learners, OpenAI Blog, 1, 9
  7. Q.Zhang M.Chen A.Bukharin et al. “Adaptive Budget Allocation for Parameter‐Efficient Fine‐Tuning ”2023 arXiv:2303.10512 [cs.CL]. · arXiv
  8. J.Zhang Y.Chen N.Niu andC.Liu “A Preliminary Evaluation of ChatGPT in Requirements Information Retrieval ”2023 arXiv Preprint arXiv:2304.12562. · DOI
  9. Ezzini S., 2023, Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 1277
  10. Luo X., 2023, Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE ‘22)
  11. Chen B., 2023, MoDRE Workshop at Requirement Engineering Conference
  12. Nahar N., 2016, Proceedings of the 2016 5th International Conference on Informatics, Electronics and Vision (ICIEV), 35
  13. Ahmad A., 2023, Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (Oulu, Finland) (EASE ‘23), New York, NY, 279
  14. Maranhão Junior J. J., 2023, AI‐Assisted Agile Software Development Workshop at XP2023 Conference
  15. T.StojanovicandS. D.Lazarević “The Application of ChatGPT for Identification of Microservices ”3 no.1(2023):99–105 Number: 1 https://ebt.rs/journals/index.php/conf‐proc/article/view/181.
  16. Gamma E., 1995, Design Patterns
  17. Viviani L., 2023, International Conference on Agile Software Development, 77
  18. 10.1109/ICSE-Companion55297.2022.9793778 · DOI · Paper page
  19. Siddiq M. L., 2023, Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language‐Based Software Engineering (NLBSE) (2023–05), 56
  20. Y.Dong J.Xue Z.Jin andG.Li “Self‐Collaboration Code Generation via ChatGPT ”2023 arXiv:2304.07590 [cs.SE]. · arXiv
  21. Jiao Sun Q., 2022, Proeedings of the 27th International Conference on Intelligent User Interfaces (New York, NY, USA, 2022‐03‐22) (IUI ‘22), 212
  22. H.Li H.Yu Y.Zhai andZ.Qian “The Hitchhiker's Guide to Program Analysis: A Journey With Large Language Models ”2023 arXiv:2308.00245 [cs.SE]. · arXiv
  23. Salza P., 2021, On the Effectiveness of Transfer Learning for Code Search, 49, no. 4, 1804
  24. Wei M., 2022, Proceedings of the 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022–05), 376
  25. Mastropaolo A., 2021, Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 336
  26. X.Liu Y.Zheng D.Zhengxiao et al. “GPT Understands Too ”(2021) arXiv:2103.10385 [cs.CL]. · arXiv
  27. R.Li L. B.Allal Y.Zi et al. “StarCoder: May the Source be With You!”(2023) arXiv:2305.06161 [cs.CL]. · arXiv
  28. Planning S., 2002, The Economic Impacts of Inadequate Infrastructure for Software Testing, National Institute of Standards and Technology, 1
  29. 10.1145/3572905 · DOI
  30. W.Ma S.Liu W.Wang et al. “The Scope of ChatGPT in Software Engineering: A Thorough Investigation ”(2023) arXiv Preprint arXiv:2305.12138. · arXiv
  31. J.White S.Hays F.Quchen J.Spencer‐Smith andD. C.Schmidt “Chatgpt Prompt Patterns for Improving Code Quality Refactoring Requirements Elicitation and Software Design ”(2023) arXiv Preprint arXiv:2303.07839. · DOI
  32. R.Feldt S.Kang J.Yoon andS.Yoo “Towards Autonomous Testing Agents via Conversational Large Language Models ”(2023) arXiv Preprint arXiv:2306.05152. · DOI · Paper page
  33. W.Sun C.Fang Y.You et al. “Automatic Code Summarization via ChatGPT: How Far Are We?”(2023) arXiv:2305.12865 [cs.SE]. · arXiv
  34. Ahmed T., 2023, Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (Rochester, MI, USA) (ASE'22)
  35. Q.Zhang C.Fang W.Sun et al. “Boosting Automated Patch Correctness Prediction via Pre‐Trained Language Model ”(2023) arXiv:2301.12453 [cs.SE]. · arXiv
  36. X.Hou Y.Zhao Y.Liu et al. “Large Language Models for Software Engineering: A Systematic Literature Review ”(2023) arXiv:2308.10620 [cs.SE]. · arXiv
  37. Sommerville I., 2016, Software Engineering
  38. Weisz J. D., 2021, Proceedings of the 26th International Conference on Intelligent User Interfaces (New York, NY, USA, 2021‐04‐14) (IUI ‘21), 402
  39. Steven I., 2023, Proceedings of the 28th International Conference on Intelligent User Interfaces (New York, NY, USA, 2023‐03‐27) (IUI ‘23), 491
  40. Anthony P., 2002, Ammeter and Janet M Dukerich. 2002. Leadership, Team Building, and Team Member Characteristics in High Performance Project Teams, Engineering Management Journal, 14, 3
  41. Dam H. K., 2019, Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE‐NIER), 41
  42. Haytham H., 2020, Elmousalami. 2020. Comparison of Artificial Intelligence Techniques for Project Conceptual Cost Prediction: A Case Study and Comparative Analysis, IEEE Transactions on Engineering Management, 68, 183
  43. OpenAI “GPT‐4 Technical Report ”(2023) arXiv:2303.08774 [cs.CL]. · arXiv
  44. D.Adiwardana M.‐T.Luong D. R.So et al. “Towards a Human‐Like Open‐Domain Chatbot. CoRR Abs/2001.09977 ”(2020) arXiv:2001.09977 https://arxiv.org/abs/2001.09977. · arXiv
  45. European Commission, 2021, EUR‐Lex ‐ 52021PC0206 ‐ EN ‐ EUR‐Lex
  46. Binkis M., 2021, Information and Software Technologies (Cham, 2021) (Communications in Computer and Information Science), 367
  47. Berrezueta‐Guzman J., 2003, Proceedings of the 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) (2023–08), 147
  48. Kazemitabaar M., 2024, Proceedings of the 23rd Koli Calling International Conference on Computing Education Research (Koli Calling ‘23), 1
  49. Balse R., 2023, Proceedings of the 16th Annual ACM India Compute Conference (COMPUTE ‘23), 49
  50. 10.48550/arXiv.2304.12269 · DOI
  51. W.Ma S.Liu W.Wang et al. “The Scope of ChatGPT in Software Engineering: A Thorough Investigation ”2023 arXiv:2305.12138 [cs] https://doi.org/10.48550/arXiv.2305.12138. · DOI
  52. F.Liu L.Yang S.Lin et al. “Exploring and Evaluating Hallucinations in LLM‐Powered Code Generation ”2024 arXiv:2404.00971 [cs] https://doi.org/10.48550/arXiv.2404.00971. · DOI · Paper page

Cited by

Sorted by citations

Metadata

Authors
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., Saari, M., Zhang, Z., Le, H., Quan, T., & Abrahamsson, P.
Affiliations
  • Norwegian University of Science and Technology (NO)
  • University of Gothenburg (SE)
  • Gdansk University of Technology Gdansk Poland
  • Monash University (AU)
  • Free University of Bozen-Bolzano (IT)
  • Austrian Post—Konzern IT Vienna Austria
  • University of Helsinki (FI)
  • Tampere University (FI)
  • Vietnam National University Ho Chi Minh City (VN)
Corresponding author
Nguyen‐Duc, Anh
Venue
Software: Practice and Experience
Publisher
Wiley
Published
2025-07-29
Volume
55
Issue
11
Pages
1806-1843
ISSN
  • 0038-0644 (ISSN-L)
  • 1097-024X
OpenAlex
W4412899734
Cited by
60
References
222
Citations by year
2026: 47 · 2025: 13
FWCI
62.39
Type
journal article
Language
en
Open access
Locations
  • Software Practice and Experience (publishedVersion)
  • Työväentutkimus Vuosikirja (submittedVersion, other-oa)
  • Chalmers Research (Chalmers University of Technology) (submittedVersion)
  • Open Research (University of Surrey) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
generative AI, software engineering, research agenda, open research questions, ChatGPT, GitHub Copilot, software development automation, decision support, requirements engineering, software design, software engineering education, software maintenance, quality assurance, dependability, transparency, sustainability, data accessibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review