CoChat

Guidelines for empirical studies in software engineering involving large language models

Baltes, Sebastian, Angermeir, Florian, Arora, Chetan, Muñoz Barón, Marvin, Chen, Chunyang, Böhme, Lukas, Calefato, Fabio, Ernst, Neil, Falessi, Davide, Fitzgerald, Brian, Fucci, Davide, He, Junda, Treude, Christoph, Kalinowski, Marcos, Lambiase, Stefano, Russo, Daniel, Lungu, Mircea, Martinez Montes, Cristina, Prechelt, Lutz, Ralph, Paul, van Tonder, Rijnard, Wagner, Stefan · 2027 · Empirical Software Engineering
Cited by 3

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape ( llm-guidelines.org ).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-026-10922-3

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 132

  1. ACM Publications Board (2020) Artifact review and badging – current. https://www.acm.org/publications/policies/artifact-review-and-badging-current, accessed 2025-01-13
  2. Agarwal R, Schwarzer M, Castro PS, Courville AC, Bellemare MG (2021) Deep reinforcement learning at the edge of the statistical precipice. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp 29304–29320. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html
  3. Ahmed T, Devanbu PT, Treude C, Pradel M (2025) Can LLMs replace manual annotation of software engineering artifacts? In: 22nd IEEE/ACM International conference on mining software repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, IEEE, pp 526–538. https://doi.org/10.1109/MSR66628.2025.00086 · DOI
  4. Ahuja S, Gumma V, Sitaram S (2024) Contamination report for multilingual benchmarks. CoRR. https://doi.org/10.48550/ARXIV.2410.16186, arXiv:2410.16186 · DOI
  5. Angermeir F, Amougou M, Kreitz M, Bauer A, Linhuber M, Fucci D, Constante FM, Méndez D, Gorschek T (2025) Reflections on the reproducibility of commercial LLM performance in empirical software engineering studies. CoRR abs/2510.25506, https://doi.org/10.48550/ARXIV.2510.25506, arXiv:2510.25506 · arXiv · doi:10.48550/arxiv.2510.25506
  6. Association for Computing Machinery (2026) ACM Policy on Authorship. https://www.acm.org/publications/policies/new-acm-policy-on-authorship. Accessed 01 June 2026
  7. Atil B, Aykent S, Chittams A, Fu L, Passonneau RJ, Radcliffe E, Rajagopal GR, Sloan A, Tudrej T, Ture F, Wu Z, Xu L, Baldwin B (2024) Non-determinism of “deterministic” LLM settings. CoRR. arXiv:2408.04667 · arXiv
  8. Austin J, Odena A, Nye MI, Bosma M, Michalewski H, Dohan D, Jiang E, Cai CJ, Terry M, Le QV, Sutton C (2021) Program synthesis with large language models. CoRR. arXiv:2108.07732 · arXiv
  9. Azanza M, Pereira J, Irastorza A, Galdos A (2024) Can LLMs facilitate onboarding software developers? an ongoing industrial case study. In: 36th International Conference on Software Engineering Education and Training, CSEE&T 2024, IEEE, pp 1–6. https://doi.org/10.1109/CSEET62301.2024.10662989 · DOI
  10. Baltes S, Ralph P (2022) Sampling in software engineering research: a critical review and guidelines. Empir Softw Eng 27(4):94. https://doi.org/10.1007/s10664-021-10072-8 · DOI · Paper page
  11. Banerjee S, Agarwal A, Singh E (2024) The vulnerability of language model benchmarks: Do they accurately reflect true LLM performance? CoRR. arXiv:2412.03597 · arXiv
  12. Bano M, Zowghi D, Whittle J (2023) Exploring qualitative research using LLMs. CoRR. https://doi.org/10.48550/ARXIV.2306.13298, arXiv:2306.13298 · DOI
  13. Bano M, Hoda R, Zowghi D, Treude C (2024) Large language models for qualitative research in software engineering: exploring opportunities and challenges. Autom Softw Eng 31(1):8. https://doi.org/10.1007/s10515-023-00407-8 · DOI
  14. Bano M, Gunatilake H, Hoda R (2025) What does a software engineer look like? exploring societal stereotypes in LLMs. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, IEEE, pp 173–184. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023 · DOI
  15. Barros CF, Azevedo BB, Neto VVG, Kassab M, Kalinowski M, do Nascimento HAD, Bandeira MCGSP (2025) Large language model for qualitative research: A systematic mapping study. In: IEEE/ACM International workshop on methodological issues with empirical studies in software engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 48–55. https://doi.org/10.1109/WSESE66602.2025.00015 · DOI
  16. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF (1996) Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276(8):637–639. https://doi.org/10.1001/jama.276.8.637 · doi:10.1001/jama.276.8.637 · Paper page
  17. Bjarnason BH, Silva A, Monperrus M (2026) On randomness in agentic evals. CoRR. https://doi.org/10.48550/ARXIV.2602.07150, arXiv:2602.07150 · DOI · Paper page
  18. Blackwell RE, Barry J, Cohn AG (2024) Towards reproducible LLM evaluation: Quantifying uncertainty in LLM benchmark scores. CoRR. https://doi.org/10.48550/ARXIV.2410.03492, arXiv:2410.03492 · DOI
  19. Borsboom D (2005) Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026 · DOI · Paper page
  20. Bouzenia I, Pradel M (2025) Understanding software engineering agents: A study of thought-action-result trajectories. In: 40th IEEE/ACM International conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, IEEE, pp 2846–2857. https://doi.org/10.1109/ASE63991.2025.00234 · DOI
  21. Cai Y, Yadavally A, Mishra A, Montejo G, Nguyen TN (2024) Programming assistant for exception handling with CodeBERT. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 94:1–94:13. https://doi.org/10.1145/3597503.3639188 · DOI
  22. Cao J, Chan YK, Ling Z, Wang W, Li S, Liu M, Qiao R, Han Y, Wang C, Yu B, He P, Wang S, Zheng Z, Lyu MR, Cheung SC (2026) Rigor, reliability, and reproducibility matter: A decade-scale survey of 572 code benchmarks. CoRR arXiv:2501.10711 · arXiv
  23. Chandra S (2025) Benchmarks for AI in software engineering (BLOG@CACM). https://cacm.acm.org/blogcacm/benchmarks-for-ai-in-software-engineering/. Accessed 24 Aug 2025
  24. Chen M, Tworek J, Jun H, Yuan Q, de Oliveira Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A, Puri R, Krueger G, Petrov M, Khlaaf H, Sastry G, Mishkin P, Chan B, Gray S, Ryder N, Pavlov M, Power A, Kaiser L, Bavarian M, Winter C, Tillet P, Such FP, Cummings D, Plappert M, Chantzis F, Barnes E, Herbert-Voss A, Guss WH, Nichol A, Paino A, Tezak N, Tang J, Babuschkin I, Balaji S, Jain S, Saunders W, Hesse C, Carr AN, Leike J, Achiam J, Misra V, Morikawa E, Radford A, Knight M, Brundage M, Murati M, Mayer K, Welinder P, McGrew B, Amodei D, McCandlish S, Sutskever I, Zaremba W (2021) Evaluating large language models trained on code. CoRR arXiv:2107.03374 · arXiv
  25. Chen Y, Ding Z, Alowain L, Chen X, Wagner DA (2023) DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In: Proceedings of the 26th international symposium on research in attacks, intrusions and defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, ACM, pp 654–668. https://doi.org/10.1145/3607199.3607242 · DOI · Paper page
  26. Chen L, Zaharia M, Zou J (2024b) How is ChatGPT’s behavior changing over time? Harvard Data Sci Rev 6(2). https://doi.org/10.1162/99608f92.5317da47 · DOI · Paper page
  27. Choi HK, Khanov M, Wei H, Li Y (2025) How contaminated is your benchmark? quantifying dataset leakage in large language models with kernel divergence. CoRR. https://doi.org/10.48550/ARXIV.2502.00678, arXiv:2502.00678 · DOI
  28. Cohen J (1992) A power primer. Psychol Bull 112(1):155–159. https://doi.org/10.1037/0033-2909.112.1.155 · doi:10.1037/0033-2909.112.1.155 · Paper page
  29. de Morais Leça M, Valença L, Santos R, de Souza Santos R (2025) Applications and implications of large language models in qualitative analysis: A new frontier for empirical software engineering. In: IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 36–43. https://doi.org/10.1109/WSESE66602.2025.00013 · DOI
  30. Dhar R, Vaidhyanathan K, Varma V (2024) Can LLMs generate architectural design decisions? - an exploratory empirical study. In: 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, IEEE, pp 79–89. https://doi.org/10.1109/ICSA59870.2024.00016 · DOI
  31. Du X, Liu M, Wang K, Wang H, Liu J, Chen Y, Feng J, Sha C, Peng X, Lou Y (2024) Evaluating large language models in class-level code generation. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 81:1–81:13. https://doi.org/10.1145/3597503.3639219 · DOI · Paper page
  32. Dybå T, Kampenes VB, Sjøberg DIK (2006) A systematic review of statistical power in software engineering experiments. Inf Softw Technol 48(8):745–755. https://doi.org/10.1016/j.infsof.2005.08.009 · DOI · Paper page
  33. El-Hajjami A, Salinesi C (2025) How good are synthetic requirements? evaluating LLM-generated datasets for AI4RE. CoRR. https://doi.org/10.48550/ARXIV.2506.21138, arXiv:2506.21138 · arXiv · doi:10.48550/arxiv.2506.21138
  34. Evertz J, Risse N, Neuer N, Müller A, Normann P, Sapia G, Gupta S, Pape D, Shaw S, Srivastav D, Wressnegger C, Quiring E, Eisenhofer T, Arp D, Schönherr L (2026) Chasing shadows: Pitfalls in LLM security research. In: 33rd Annual Network and Distributed System Security Symposium, NDSS 2026, San Diego, California, USA, February 23-27, 2026, The Internet Society, https://www.ndss-symposium.org/ndss-paper/chasing-shadows-pitfalls-in-llm-security-research/ · DOI
  35. Gallegos IO, Rossi RA, Barrow J, Tanjim MM, Kim S, Dernoncourt F, Yu T, Zhang R, Ahmed NK (2024) Bias and fairness in large language models: A survey. Comput Linguist 50(3):1097–1179. https://doi.org/10.1162/COLI_A_00524 · DOI · Paper page
  36. Galster M, Mohsenimofidi S, Lulla JL, Abubakar MA, Treude C, Baltes S (2026) Configuring agentic AI coding tools: An exploratory study. In: Proceedings of the 3rd ACM international conference on AI-powered software (AIware 2026) · DOI
  37. Gerosa MA, Trinkenreich B, Steinmacher I, Sarma A (2024) Can AI serve as a substitute for human subjects in software engineering research? Autom Softw Eng 31(1):13. https://doi.org/10.1007/s10515-023-00409-6 · DOI
  38. Gibney E (2024) Not all ‘open source’ AI models are actually open. Nature News. https://doi.org/10.1038/d41586-024-02012-5 · DOI
  39. Golnari PA, Kumarappan A, Wen W, Liu X, Ryan G, Sun Y, Fu S, Nallipogu E (2026) DevBench: A realistic, developer-informed benchmark for code generation models. CoRR. https://doi.org/10.48550/ARXIV.2601.11895, arXiv:2601.11895 · DOI
  40. Gonçalves J, Silva M, Cabral B, Dias T, Maia E, Praça I, Severino R, Ferreira LL (2025) Evaluating LLaMA 3.2 for software vulnerability detection. In: Praça I, Bernardi S, Inácio PRM (eds) Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, Springer, Communications in Computer and Information Science, pp 38–51. https://doi.org/10.1007/978-3-031-94855-8_3 · DOI
  41. Guo D, Zhu Q, Yang D, Xie Z, Dong K, Zhang W, Chen G, Bi X, Wu Y, Li YK, Luo F, Xiong Y, Liang W (2024) DeepSeek-Coder: When the large language model meets programming - the rise of code intelligence. CoRR. https://doi.org/10.48550/ARXIV.2401.14196, arXiv:2401.14196 · DOI · Paper page
  42. Harding J, D’Alessandro W, Laskowski NG, Long R (2024) AI language models cannot replace human research participants. AI Soc 39(5):2603–2605. https://doi.org/10.1007/s00146-023-01725-x · DOI
  43. He J, Rungta M, Koleczek D, Sekhon A, Wang FX, Hasan S (2024a) Does prompt formatting have any impact on LLM performance? CoRR. https://doi.org/10.48550/ARXIV.2411.10541, arXiv:2411.10541 · DOI
  44. He Z, Huang C, Ding CC, Rohatgi S, Huang TK (2024b) If in a crowdsourced data annotation pipeline, a GPT-4. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI conference on human factors in computing systems, CHI 2024, ACM, pp 1040:1–1040:25. https://doi.org/10.1145/3613904.3642834 · DOI
  45. Holtzman A, Buys J, Du L, Forbes M, Choi Y (2020) The curious case of neural text degeneration. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH
  46. Hosking T, Blunsom P, Bartolo M (2024) Human feedback is not gold standard. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=7W3GLNImfS
  47. Hu X, Niu F, Chen J, Zhou X, Zhang J, He J, Xia X, Lo D (2025) Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. CoRR. https://doi.org/10.48550/ARXIV.2505.08903, arXiv:2505.08903 · arXiv · doi:10.48550/arxiv.2505.08903
  48. Huang F, Kwak H, An J (2023) Is ChatGPT better than human annotators? potential and limitations of ChatGPT in explaining implicit hate speech. In: Ding Y, Tang J, Sequeda JF, Aroyo L, Castillo C, Houben G (eds) Companion Proceedings of the ACM Web Conference 2023, WWW 2023, ACM, pp 294–297. https://doi.org/10.1145/3543873.3587368 · DOI · Paper page
  49. Huang J, Yi B, Sun W, Wan B, Xu Y, Feng Y, Ye W, Qin Q (2024) Enhancing review classification via LLM-based data annotation and multi-perspective feature representation learning. SSRN Electronic Journal pp 1–15. https://doi.org/10.2139/ssrn.5002351, https://ssrn.com/abstract=5002351 · DOI
  50. Hui B, Yang J, Cui Z, Yang J, Liu D, Zhang L, Liu T, Zhang J, Yu B, Dang K, Yang A, Men R, Huang F, Ren X, Ren X, Zhou J, Lin J (2024) Qwen2.5-Coder technical report. CoRR. https://doi.org/10.48550/ARXIV.2409.12186, arXiv:2409.12186 · DOI · Paper page
  51. Hymel C, Johnson H (2025) Analysis of LLMs vs human experts in requirements engineering. CoRR. https://doi.org/10.48550/ARXIV.2501.19297, arXiv:2501.19297 · DOI
  52. Jahic J, Sami A (2024) State of practice: LLMs in software engineering and software architecture. In: 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, IEEE, pp 311–318. https://doi.org/10.1109/ICSA-C63560.2024.00059 · DOI
  53. Jain N, Han K, Gu A, Li W, Yan F, Zhang T, Wang S, Solar-Lezama A, Sen K, Stoica I (2024) LiveCodeBench: Holistic and contamination free evaluation of large language models for code. CoRR. https://doi.org/10.48550/ARXIV.2403.07974, arXiv:2403.07974 · DOI · Paper page
  54. Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR (2024) SWE-bench: Can language models resolve real-world github issues? In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66
  55. Jowsey T, Braun V, Clarke V, Lupton D, Fine M (2025) We reject the use of generative artificial intelligence for reflexive qualitative research. Qual Inq. https://doi.org/10.1177/10778004251401851 · DOI
  56. Kapoor S, Cantrell EM, Peng K, Pham TH, Bail CA, Gundersen OE, Hofman JM, Hullman J, Lones MA, Malik MM, Nanayakkara P, Poldrack RA, Raji ID, Roberts M, Salganik MJ, Serra-Garcia M, Stewart BM, Vandewiele G, Narayanan A (2024) REFORMS: Consensus-based recommendations for machine-learning-based science. Sci Adv 10(18):eadk3452. https://doi.org/10.1126/sciadv.adk3452 · DOI · Paper page
  57. Khojah R, Mohamad M, Leitner P, de Oliveira Neto FG (2024) Beyond code generation: An observational study of ChatGPT usage in software engineering practice. Proc ACM Softw Eng 1(FSE):1819–1840. https://doi.org/10.1145/3660788 · DOI · Paper page
  58. Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K (2024) Can large language models replace humans in systematic reviews? evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods 15(4):616–626. https://doi.org/10.1002/jrsm.1715, https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1715 · DOI · Paper page
  59. Kirk R, Mediratta I, Nalmpantis C, Luketina J, Hambro E, Grefenstette E, Raileanu R (2023) Understanding the effects of RLHF on LLM generalisation and diversity. CoRR. arXiv:2310.06452 · arXiv
  60. Kong A, Zhao S, Chen H, Li Q, Qin Y, Sun R, Zhou X, Wang E, Dong X (2024) Better zero-shot reasoning with role-play prompting. In: Duh K, Gómez-Adorno H, Bethard S (eds) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, Association for Computational Linguistics, pp 4099–4113. https://doi.org/10.18653/V1/2024.NAACL-LONG.228 · DOI · Paper page
  61. Korn A, Zaruchas L, Arora C, Metzger A, Smolka S, Wang F, Vogelsang A (2026) Reporting LLM prompting in automated software engineering: A guideline based on current practices and expectations. CoRR. https://doi.org/10.48550/ARXIV.2601.01954, to appear at FORGE 2026, co-located with ICSE 2026, Rio de Janeiro, Brazil. arXiv:2601.01954 · arXiv · doi:10.48550/arxiv.2601.01954
  62. Kübler JM, Budhathoki K, Kleindessner M, Zhou X, Yin J, Khetan A, Karypis G (2026) When LLMs get significantly worse: A statistical approach to detect model degradations. In: Proceedings of the International Conference on Learning Representations (ICLR)
  63. Lago P, Runeson P, Song Q, Verdecchia R (2024) Threats to validity in software engineering - hypocritical paper section or essential analysis? In: Franch X, Daneva M, Martínez-Fernández S, Quaranta L (eds) Proceedings of the 18th ACM/IEEE international symposium on Empirical Software Engineering and Measurement, ESEM 2024, Barcelona, Spain, October 24-25, 2024, ACM, pp 314–324. https://doi.org/10.1145/3674805.3686691 · DOI
  64. Li D, Gupta K, Bhaduri M, Sathiadoss P, Bhatnagar S, Chong J (2024) Comparing GPT-3.5 and GPT-4 accuracy and drift in radiology diagnosis please cases. Radiology 310(1):e232411. https://doi.org/10.1148/radiol.232411 · DOI
  65. Liu J, Xia CS, Wang Y, Zhang L (2023) Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In: Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S (eds) Advances in neural information processing systems 36: Annual conference on neural information processing systems 2023, NeurIPS 2023, http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
  66. Liu M, Huang X, He W, Xie Y, Zhang JM, Jing X, Chen Z, Ma Y (2024a) Research artifacts in software engineering publications: Status and trends. J Syst Softw 213:112032. https://doi.org/10.1016/j.jss.2024.112032 · DOI · Paper page
  67. Liu X, Yu H, Zhang H, Xu Y, Lei X, Lai H, Gu Y, Ding H, Men K, Yang K, Zhang S, Deng X, Zeng A, Du Z, Zhang C, Shen S, Zhang T, Su Y, Sun H, Huang M, Dong Y, Tang J (2024b) AgentBench: Evaluating LLMs as agents. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=zAdUB0aCTQ
  68. Lubos S, Felfernig A, Tran TNT, Garber D, Mansi ME, Erdeniz SP, Le V (2024) Leveraging LLMs for the quality assurance of software requirements. In: Liebel G, Hadar I, Spoletini P (eds) 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, IEEE, pp 389–397. https://doi.org/10.1109/RE59067.2024.00046 · DOI
  69. Madampe K, Grundy J, Hoda R, Obie HO (2024) The struggle is real! the agony of recruiting participants for empirical software engineering studies. In: 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, IEEE, pp 417–422. https://doi.org/10.1109/VL/HCC60511.2024.00065 · DOI
  70. Martinez Montes C, Feldt R, Martos CM, Ouhbi S, Premanandan S, Graziotin D (2025) Large language models in thematic analysis: Prompt engineering, evaluation, and guidelines for qualitative software engineering research. CoRR. https://doi.org/10.48550/ARXIV.2510.18456, arXiv:2510.18456 · arXiv · doi:10.48550/arxiv.2510.18456
  71. Menzies T (2025) The case for compact AI. Commun ACM 68(9):6–7. https://doi.org/10.1145/3746057 · DOI
  72. Mitu NE, Mitu GT (2024) The hidden cost of AI: Carbon footprint and mitigation strategies. Revista de Stiinte Politice Revue des Sci Politiq 84:9–16. https://doi.org/10.2139/ssrn.5036344 · doi:10.2139/ssrn.5036344
  73. Mohamed S, Parvin A, Parra E (2024) Chatting with AI: deciphering developer conversations with ChatGPT. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International conference on mining software repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 187–191. https://doi.org/10.1145/3643991.3645078 · DOI
  74. Moumoula MB, Kaboré AK, Klein J, Bissyandé TF (2024) Large language models for cross-language code clone detection. CoRR. https://doi.org/10.48550/ARXIV.2408.04430, arXiv:2408.04430 · arXiv · doi:10.48550/arxiv.2408.04430
  75. Navarro KF, Syriani E, Arawjo I (2026) Reporting and reviewing LLM-integrated systems in HCI: challenges and considerations. In: Oliver N, Shamma DA, Candello H, César P, Lopes P, Bozzon A, Kosch T, Liao VQ, Ma X, Artizzu V, Draxler F, López G, Reinschluessel AV, Tong X, Dugas POT (eds) Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026, Barcelona, Spain, April 13-17, 2026, ACM, pp 1546:1–1546:18. https://doi.org/10.1145/3772318.3790439 · DOI
  76. Ornelas T, Araújo AA, Araújo J, Araújo M, Trinkenreich B, Kalinowski M (2025) LLM-assisted thematic analysis: Opportunities, limitations, and recommendations. CoRR. https://doi.org/10.48550/ARXIV.2511.14528 , arXiv:2511.14528 · arXiv · doi:10.48550/arxiv.2511.14528
  77. Pangakis N, Wolken S, Fasching N (2023) Automated annotation with generative AI requires validation. CoRR. https://doi.org/10.48550/ARXIV.2306.00176, arXiv:2306.00176 · DOI · Paper page
  78. Panickssery A, Bowman SR, Feng S (2024) LLM evaluators recognize and favor their own generations. In: Globersons A, Mackey L, Belgrave D, Fan A, Paquet U, Tomczak JM, Zhang C (eds) Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. http://papers.nips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html
  79. Papineni K, Roukos S, Ward T, Zhu W (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, pp 311–318, https://doi.org/10.3115/1073083.1073135, https://aclanthology.org/P02-1040/ · DOI · Paper page
  80. Pezeshkpour P, Hruschka E (2024) Large language models sensitivity to the order of options in multiple-choice questions. In: Duh K, Gómez-Adorno H, Bethard S (eds) Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, Association for Computational Linguistics, Findings of ACL, vol NAACL 2024, pp 2006–2017. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130 · DOI · Paper page
  81. Pineau J, Vincent-Lamarre P, Sinha K, Larivière V, Beygelzimer A, d’Alché-Buc F, Fox EB, Larochelle H (2021) Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). J Mach Learn Res 22:164:1–164:20. https://jmlr.org/papers/v22/20-303.html
  82. Rabbi MF, Champa AI, Zibran MF, Islam MR (2024) AI writes, we analyze: The ChatGPT Python code saga. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 177–181. https://doi.org/10.1145/3643991.3645076 · DOI
  83. Ralph P, Tempero ED (2018) Construct validity in software engineering research and software metrics. In: Rainer A, MacDonell SG, Keung JW (eds) Proceedings of the 22nd international conference on Evaluation and Assessment in Software Engineering, EASE2018, ACM, pp 13–23. https://doi.org/10.1145/3210459.3210461 · DOI · Paper page
  84. Ralph P, bin Ali N, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A, de França BBN, Furia CA, Gay G, Gold N, Graziotin D, He P, Hoda R, Juristo N, Kitchenham B, Lenarduzzi V, Martínez J, Melegati J, Mendez D, Menzies T, Molleri J, Pfahl D, Robbes R, Russo D, Saarimäki N, Sarro F, Taibi D, Siegmund J, Spinellis D, Staron M, Stol K, Storey MA, Taibi D, Tamburri D, Torchiano M, Treude C, Turhan B, Wang X, Vegas S (2021) Empirical standards for software engineering research. arXiv:2010.03525 · arXiv
  85. Ralph P, Kuutila M, Arif H, Ayoola B (2024) Teaching software metrology: The science of measurement for software engineering. In: Méndez D, Avgeriou P, Kalinowski M, Ali NB (eds) Handbook on teaching empirical software engineering, Springer Nature Switzerland, pp 101–154. https://doi.org/10.1007/978-3-031-71769-7_5 · DOI
  86. Rasheed Z, Waseem M, Ahmad A, Kemell K, Wang X, Nguyen-Duc A, Abrahamsson P (2024) Can large language models serve as data analysts? A multi-agent assisted approach for qualitative data analysis. CoRR. https://doi.org/10.48550/ARXIV.2402.01386, arXiv:2402.01386 · DOI
  87. Reiss MV (2023) Testing the reliability of ChatGPT for text annotation and classification: A cautionary remark. CoRR. https://doi.org/10.48550/ARXIV.2304.11085, arXiv:2304.11085 · DOI · Paper page
  88. Renze M (2024) The effect of sampling temperature on problem solving in large language models. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp 7346–7356. https://doi.org/10.18653/v1/2024.findings-emnlp.432 · DOI · Paper page
  89. Richards J, Wessel M (2024) What you need is what you get: Theory of mind for an LLM-based code understanding assistant. In: IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, IEEE, pp 666–671, https://doi.org/10.1109/ICSME58944.2024.00070 · DOI
  90. Robillard MP, Arya DM, Ernst NA, Guo JLC, Lamothe M, Nassif M, Novielli N, Serebrenik A, Steinmacher I, Stol K (2024) Communicating study design trade-offs in software engineering. ACM Trans Softw Eng Methodol 33(5):112:1-112:10. https://doi.org/10.1145/3649598 · DOI
  91. Ronanki K, Berger C, Horkoff J (2023) Investigating ChatGPT’s potential to assist in requirements elicitation processes. In: 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, pp 354–361 · DOI
  92. Rozière B, Lachaux M, Chanussot L, Lample G (2020) Unsupervised translation of programming languages. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. https://proceedings.neurips.cc/paper/2020/hash/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html
  93. Rozière B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, Adi Y, Liu J, Remez T, Rapin J, Kozhevnikov A, Evtimov I, Bitton J, Bhatt M, Canton-Ferrer C, Grattafiori A, Xiong W, Défossez A, Copet J, Azhar F, Touvron H, Martin L, Usunier N, Scialom T, Synnaeve G (2023) Code Llama: Open foundation models for code. CoRR. https://doi.org/10.48550/ARXIV.2308.12950, arXiv:2308.12950 · DOI · Paper page
  94. Russo D (2024) Navigating the complexity of generative AI adoption in software engineering. ACM Trans Softw Eng Methodol 33(5):135:1-135:50. https://doi.org/10.1145/3652154 · DOI · Paper page
  95. Sallou J, Durieux T, Panichella A (2024) Breaking the silence: the threats of using LLMs in software engineering. In: Proceedings of the 2024 ACM/IEEE 44th international conference on software engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 102–106. https://doi.org/10.1145/3639476.3639764 · DOI
  96. Santos A, Vegas S, Oivo M, Juristo N (2021) A procedure and guidelines for analyzing groups of software engineering replications. IEEE Trans Software Eng 47(9):1742–1763. https://doi.org/10.1109/TSE.2019.2935720 · DOI
  97. Schäfer M, Nadi S, Eghbali A, Tip F (2024) An empirical evaluation of using large language models for automated unit test generation. IEEE Trans Software Eng 50(1):85–105. https://doi.org/10.1109/TSE.2023.3334955 · DOI · Paper page
  98. Schilke O, Reimann M (2025) The transparency dilemma: How AI disclosure erodes trust. Organ Behav Hum Decis Process 188:104405. https://doi.org/10.1016/j.obhdp.2025.104405 , https://www.sciencedirect.com/science/article/pii/S0749597825000172 · DOI · Paper page
  99. Schneider K, Fotrousi F, Wohlrab R (2025) A reference model for empirically comparing LLMs with humans. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025. IEEE, pp 130–134. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018 · DOI
  100. Schröder S, Morgenroth T, Kuhl U, Vaquet V, Paaßen B (2025) Large language models do not simulate human psychology. CoRR. https://doi.org/10.48550/ARXIV.2508.06950, arXiv:2508.06950 · DOI
  101. Schroeder K, Wood-Doughty Z (2024) Can you trust LLM judgments? reliability of LLM-as-a-judge. CoRR. https://doi.org/10.48550/ARXIV.2412.12509, arXiv:2412.12509 · DOI · Paper page
  102. Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332. https://doi.org/10.1136/bmj.c332 · DOI · Paper page
  103. Sclar M, Choi Y, Tsvetkov Y, Suhr A (2024) Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT
  104. Shull F, Singer J, Sjøberg DIK (eds) (2008) Guide to advanced empirical software engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5 · DOI · Paper page
  105. Silva A, Monperrus M (2025) RepairBench: Leaderboard of frontier models for program repair. In: IEEE/ACM International workshop on large language models for code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 9–16. https://doi.org/10.1109/LLM4CODE66737.2025.00006 · DOI
  106. Sjøberg DIK, Bergersen GR (2023) Construct validity in software engineering. IEEE Trans Software Eng 49(3):1374–1396. https://doi.org/10.1109/TSE.2022.3176725 · DOI
  107. Song Y, Wang G, Li S, Lin BY (2025) The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. In: Chiruzzo L, Ritter A, Wang L (eds) Proceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: Human language technologies, NAACL 2025 - Volume 1: Long Papers, Association for Computational Linguistics, pp 4195–4206. https://doi.org/10.18653/V1/2025.NAACL-LONG.211 · DOI
  108. Sonnekalb T, Gruner B, Brust C, Mäder P (2022) Generalizability of code clone detection on CodeBERT. In: 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, ACM, pp 143:1–143:3. https://doi.org/10.1145/3551349.3561165 · DOI
  109. Sullivan GM, Sargeant J (2011) Qualities of qualitative research: part I. J Grad Med Educ 3(4):449–452 · DOI
  110. Sumers TR, Yao S, Narasimhan K, Griffiths TL (2024) Cognitive architectures for language agents. Trans Mach Learn Res 2024. https://openreview.net/forum?id=1i6ZCvflQJ
  111. Takerngsaksiri W, Pasuksmit J, Thongtanunam P, Tantithamthavorn C, Zhang R, Jiang F, Li J, Cook E, Chen K, Wu M (2025) Human-in-the-loop software development agents. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, IEEE, pp 342–352. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036 · DOI · Paper page
  112. Verdecchia R, Engström E, Lago P, Runeson P, Song Q (2023) Threats to validity in software engineering research: A critical reflection. Inf Softw Technol 164:107329. https://doi.org/10.1016/j.infsof.2023.107329 · DOI
  113. von Kistowski J, Arnold JA, Huppler K, Lange K, Henning JL, Cao P (2015) How to build a benchmark. In: John LK, Smith CU, Sachs K, Lladó CM (eds) Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, Austin, TX, USA, January 31 - February 4, 2015, ACM, pp 333–336. https://doi.org/10.1145/2668930.2688819 · DOI · Paper page
  114. Wagner S, Muñoz Barón M, Falessi D, Baltes S (2025) Towards evaluation guidelines for empirical studies involving LLMs. In: IEEE/ACM International workshop on methodological issues with empirical studies in software engineering, WSESE@ICSE 2025, May 3, 2025, IEEE, pp 24–27. https://doi.org/10.1109/WSESE66602.2025.00011 · DOI
  115. Wang S, Liu Y, Xu Y, Zhu C, Zeng M (2021) Want to reduce labeling cost? GPT-3 can help. In: Moens M, Huang X, Specia L, Yih SW (eds) Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, Association for Computational Linguistics, Findings of ACL, vol EMNLP 2021, pp 4195–4205. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354 · DOI · Paper page
  116. Wang C, Huang K, Zhang J, Feng Y, Zhang L, Liu Y, Peng X (2024a) How and why LLMs use deprecated APIs in code completion? an empirical study. CoRR. https://doi.org/10.48550/ARXIV.2406.09834 , arXiv:2406.09834 · DOI
  117. Wang X, Kim H, Rahman S, Mitra K, Miao Z (2024b) Human-LLM collaborative annotation through effective verification of LLM labels. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI conference on human factors in computing systems, CHI 2024, ACM, pp 303:1–303:21. https://doi.org/10.1145/3613904.3641960 · DOI · Paper page
  118. Wang A, Morgenstern J, Dickerson JP (2025a) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat Mac Intell 7(3):400–411. https://doi.org/10.1038/s42256-025-00986-z · DOI
  119. Wang F, Arora C, Liu Y, Huang K, Tantithamthavorn C, Aleti A, Sambathkumar D, Lo D (2025b) Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs. In: 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, pp 3334–3345. https://doi.org/10.1109/ASE63991.2025.00275 · DOI
  120. Widder DG, Whittaker M, West SM (2024) Why ‘open’AI systems are actually closed, and why this matters. Nature 635(8040):827–833 · DOI · Paper page
  121. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2024) Experimentation in software engineering, 2nd edn. Springer. https://doi.org/10.1007/978-3-662-69306-3 · DOI · Paper page
  122. Xia Y, Shao H, Deng X (2024) VulCoBERT: A CodeBERT-based system for source code vulnerability detection. In: 2024 International conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024, ACM. https://doi.org/10.1145/3665348.3665391 · DOI
  123. Xiao T, Treude C, Hata H, Matsumoto K (2024) DevGPT: Studying developer-ChatGPT conversations. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 227–230. https://doi.org/10.1145/3643991.3648400 · DOI · Paper page
  124. Xu C, Guan S, Greene D, Kechadi MT (2024a) Benchmark data contamination of large language models: A survey. CoRR. https://doi.org/10.48550/ARXIV.2406.04244, arXiv:2406.04244 · DOI · Paper page
  125. Xu R, Sun Y, Ren M, Guo S, Pan R, Lin H, Sun L, Han X (2024b) AI for social science and social science of AI: A survey. Inf Process Manag 61(2):103665. https://doi.org/10.1016/J.IPM.2024.103665 · DOI
  126. Yan L, Hwang A, Wu Z, Head A (2024) Ivie: Lightweight anchored explanations of just-generated code. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, ACM, pp 140:1–140:15. https://doi.org/10.1145/3613904.3642239 · DOI
  127. Yang G, Zhou Y, Chen X, Zhang X, Han T, Chen T (2023) ExploitGen: Template-augmented exploit code generation based on CodeBERT. J Syst Softw 197:111577. https://doi.org/10.1016/J.JSS.2022.111577 · DOI
  128. Yang Y, Shin J, Choi B, Park M, Ju D, Lee C, Chun S, Kang D, Kim J, Yoon S (2025) From code foundation models to agents and applications: A comprehensive survey and practical guide to code intelligence. arXiv preprint arXiv:2502.11827 · arXiv
  129. Zhang J, Yu S, Chong D, Sicilia A, Tomz MR, Manning CD, Shi W (2025) Verbalized sampling: How to mitigate mode collapse and unlock LLM diversity. CoRR. arXiv:2510.01171 · arXiv
  130. Zhao C, Habule M, Zhang W (2025) Large language models (LLMs) as research subjects: Status, opportunities and challenges. New Ideas Psychol 79:101167. https://doi.org/10.1016/j.newideapsych.2025.101167, https://www.sciencedirect.com/science/article/pii/S0732118X25000236 · DOI
  131. Zhou R, Chen L, Yu K (2024) Is LLM a reliable reviewer? A comprehensive evaluation of LLM on automatic paper reviewing tasks. In: Calzolari N, Kan M, Hoste V, Lenci A, Sakti S, Xue N (eds) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, ELRA and ICCL, pp 9340–9351. https://aclanthology.org/2024.lrec-main.816 · DOI · Paper page
  132. Zhu Y, Zhang P, ul Haq E, Hui P, Tyson G (2023) Can ChatGPT reproduce human-generated labels? A study of social computing tasks. CoRR. https://doi.org/10.48550/ARXIV.2304.10145, arXiv:2304.10145 · arXiv · doi:10.48550/arxiv.2304.10145 · Paper page

Metadata

Authors
Baltes, S., Angermeir, F., Arora, C., Muñoz Barón, M., Chen, C., Böhme, L., Calefato, F., Ernst, N., Falessi, D., Fitzgerald, B., Fucci, D., He, J., Treude, C., Kalinowski, M., Lambiase, S., Russo, D., Lungu, M., Martinez Montes, C., Prechelt, L., Ralph, P., van Tonder, R., & Wagner, S.
Venue
Empirical Software Engineering
Published
2026-11-03
Volume
32
Issue
2
ISSN
  • 1382-3256
  • 1573-7616
Cited by
3
References
31
Language
en
Licence
Funders
Medizinische Fakultät Heidelberg der Universität Heidelberg

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.