Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 3

The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.

Guidelines for empirical studies in software engineering involving large language models

Baltes, Sebastian, Angermeir, Florian, Arora, Chetan, Muñoz Barón, Marvin, Chen, Chunyang, Böhme, Lukas, Calefato, Fabio, Ernst, Neil, Falessi, Davide, Fitzgerald, Brian, Fucci, Davide, He, Junda, Treude, Christoph, Kalinowski, Marcos, Lambiase, Stefano, Russo, Daniel, Lungu, Mircea, Martinez Montes, Cristina, Prechelt, Lutz, Ralph, Paul, van Tonder, Rijnard, Wagner, Stefan·2027·Empirical Software Engineering
DOI 10.1007/s10664-026-10922-3
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape ( llm-guidelines.org ).

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10664-026-10922-3

Cites

16 of 16 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 16 references

Research works printed but not linked: 145

  1. ACM Publications Board (2020) Artifact review and badging – current. https://www.acm.org/publications/policies/artifact-review-and-badging-current, accessed 2025-01-13
  2. Agarwal R, Schwarzer M, Castro PS, Courville AC, Bellemare MG (2021) Deep reinforcement learning at the edge of the statistical precipice. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in neural information processing systems 34: annual conference on neural information processing systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp 29304–29320. https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html
  3. Ahmed N, Wahed M, Thompson NC (2023) The growing influence of industry in AI research. Science 379(6635):884–886. https://doi.org/10.1126/science.ade2420 · DOI · Paper page
  4. Ahmed T, Devanbu PT, Treude C, Pradel M (2025) Can LLMs replace manual annotation of software engineering artifacts? In: 22nd IEEE/ACM International conference on mining software repositories, MSR@ICSE 2025, Ottawa, ON, Canada, April 28-29, 2025, IEEE, pp 526–538. https://doi.org/10.1109/MSR66628.2025.00086 · DOI
  5. Ahuja S, Gumma V, Sitaram S (2024) Contamination report for multilingual benchmarks. CoRR. https://doi.org/10.48550/ARXIV.2410.16186, arXiv:2410.16186 · DOI
  6. Angermeir F, Amougou M, Kreitz M, Bauer A, Linhuber M, Fucci D, Constante FM, Méndez D, Gorschek T (2025) Reflections on the reproducibility of commercial LLM performance in empirical software engineering studies. CoRR abs/2510.25506, https://doi.org/10.48550/ARXIV.2510.25506, arXiv:2510.25506 · arXiv · doi:10.48550/arxiv.2510.25506
  7. Association for Computing Machinery (2026) ACM Policy on Authorship. https://www.acm.org/publications/policies/new-acm-policy-on-authorship. Accessed 01 June 2026
  8. Atil B, Aykent S, Chittams A, Fu L, Passonneau RJ, Radcliffe E, Rajagopal GR, Sloan A, Tudrej T, Ture F, Wu Z, Xu L, Baldwin B (2024) Non-determinism of “deterministic” LLM settings. CoRR. arXiv:2408.04667 · arXiv
  9. Austin J, Odena A, Nye MI, Bosma M, Michalewski H, Dohan D, Jiang E, Cai CJ, Terry M, Le QV, Sutton C (2021) Program synthesis with large language models. CoRR. arXiv:2108.07732 · arXiv
  10. Azanza M, Pereira J, Irastorza A, Galdos A (2024) Can LLMs facilitate onboarding software developers? an ongoing industrial case study. In: 36th International Conference on Software Engineering Education and Training, CSEE&T 2024, IEEE, pp 1–6. https://doi.org/10.1109/CSEET62301.2024.10662989 · DOI
  11. Baltes S, Ralph P (2022) Sampling in software engineering research: a critical review and guidelines. Empir Softw Eng 27(4):94. https://doi.org/10.1007/s10664-021-10072-8 · DOI · Paper page
  12. Banerjee S, Agarwal A, Singh E (2024) The vulnerability of language model benchmarks: Do they accurately reflect true LLM performance? CoRR. arXiv:2412.03597 · arXiv
  13. Bano M, Zowghi D, Whittle J (2023) Exploring qualitative research using LLMs. CoRR. https://doi.org/10.48550/ARXIV.2306.13298, arXiv:2306.13298 · DOI
  14. Bano M, Hoda R, Zowghi D, Treude C (2024) Large language models for qualitative research in software engineering: exploring opportunities and challenges. Autom Softw Eng 31(1):8. https://doi.org/10.1007/s10515-023-00407-8 · DOI
  15. Bano M, Gunatilake H, Hoda R (2025) What does a software engineer look like? exploring societal stereotypes in LLMs. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, IEEE, pp 173–184. https://doi.org/10.1109/ICSE-SEIS66351.2025.00023 · DOI
  16. Barros CF, Azevedo BB, Neto VVG, Kassab M, Kalinowski M, do Nascimento HAD, Bandeira MCGSP (2025) Large language model for qualitative research: A systematic mapping study. In: IEEE/ACM International workshop on methodological issues with empirical studies in software engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 48–55. https://doi.org/10.1109/WSESE66602.2025.00015 · DOI
  17. Bavaresco A, Bernardi R, Bertolazzi L, Elliott D, Fernández R, Gatt A, Ghaleb E, Giulianelli M, Hanna M, Koller A, Martins AFT, Mondorf P, Neplenbroek V, Pezzelle S, Plank B, Schlangen D, Suglia A, Surikuchi AK, Takmaz E, Testoni A (2024) LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks. CoRR. https://doi.org/10.48550/ARXIV.2406.18403, arXiv:2406.18403 · DOI · Paper page
  18. Bean AM, Kearns RO, Romanou A, Hafner FS, Mayne H, Batzner J, Foroutan N, Schmitz C, Korgul K, Batra H, Deb O, Beharry E, Emde C, Foster T, Gausen A, Grandury M, Han S, Hofmann V, Ibrahim L, Kim H, Kirk HR, Lin F, Liu GKM, Luettgau L, Magomere J, Rystrøm J, Sotnikova A, Yang Y, Zhao Y, Bibi A, Bosselut A, Clark R, Cohan A, Foerster J, Gal Y, Hale SA, Raji ID, Summerfield C, Torr PHS, Ududec C, Rocher L, Mahdi A (2025) Measuring what matters: Construct validity in large language model benchmarks. In: Advances in neural information processing systems 38: Annual conference on neural information processing systems 2025, NeurIPS 2025, Datasets and Benchmarks Track. arXiv:2511.04703 · DOI · Paper page
  19. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF (1996) Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276(8):637–639. https://doi.org/10.1001/jama.276.8.637 · doi:10.1001/jama.276.8.637 · Paper page
  20. Bjarnason BH, Silva A, Monperrus M (2026) On randomness in agentic evals. CoRR. https://doi.org/10.48550/ARXIV.2602.07150, arXiv:2602.07150 · DOI · Paper page
  21. Blackwell RE, Barry J, Cohn AG (2024) Towards reproducible LLM evaluation: Quantifying uncertainty in LLM benchmark scores. CoRR. https://doi.org/10.48550/ARXIV.2410.03492, arXiv:2410.03492 · DOI
  22. Borsboom D (2005) Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press. https://doi.org/10.1017/CBO9780511490026 · DOI · Paper page
  23. Bouzenia I, Pradel M (2025) Understanding software engineering agents: A study of thought-action-result trajectories. In: 40th IEEE/ACM International conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, IEEE, pp 2846–2857. https://doi.org/10.1109/ASE63991.2025.00234 · DOI
  24. Cai Y, Yadavally A, Mishra A, Montejo G, Nguyen TN (2024) Programming assistant for exception handling with CodeBERT. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 94:1–94:13. https://doi.org/10.1145/3597503.3639188 · DOI
  25. Cao J, Chan YK, Ling Z, Wang W, Li S, Liu M, Qiao R, Han Y, Wang C, Yu B, He P, Wang S, Zheng Z, Lyu MR, Cheung SC (2026) Rigor, reliability, and reproducibility matter: A decade-scale survey of 572 code benchmarks. CoRR arXiv:2501.10711 · arXiv
  26. Chandra S (2025) Benchmarks for AI in software engineering (BLOG@CACM). https://cacm.acm.org/blogcacm/benchmarks-for-ai-in-software-engineering/. Accessed 24 Aug 2025
  27. Chen M, Tworek J, Jun H, Yuan Q, de Oliveira Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A, Puri R, Krueger G, Petrov M, Khlaaf H, Sastry G, Mishkin P, Chan B, Gray S, Ryder N, Pavlov M, Power A, Kaiser L, Bavarian M, Winter C, Tillet P, Such FP, Cummings D, Plappert M, Chantzis F, Barnes E, Herbert-Voss A, Guss WH, Nichol A, Paino A, Tezak N, Tang J, Babuschkin I, Balaji S, Jain S, Saunders W, Hesse C, Carr AN, Leike J, Achiam J, Misra V, Morikawa E, Radford A, Knight M, Brundage M, Murati M, Mayer K, Welinder P, McGrew B, Amodei D, McCandlish S, Sutskever I, Zaremba W (2021) Evaluating large language models trained on code. CoRR arXiv:2107.03374 · arXiv
  28. Chen Y, Ding Z, Alowain L, Chen X, Wagner DA (2023) DiverseVul: A new vulnerable source code dataset for deep learning based vulnerability detection. In: Proceedings of the 26th international symposium on research in attacks, intrusions and defenses, RAID 2023, Hong Kong, China, October 16-18, 2023, ACM, pp 654–668. https://doi.org/10.1145/3607199.3607242 · DOI
  29. Chen L, Zaharia M, Zou J (2024b) How is ChatGPT’s behavior changing over time? Harvard Data Sci Rev 6(2). https://doi.org/10.1162/99608f92.5317da47 · DOI · Paper page
  30. Choi HK, Khanov M, Wei H, Li Y (2025) How contaminated is your benchmark? quantifying dataset leakage in large language models with kernel divergence. CoRR. https://doi.org/10.48550/ARXIV.2502.00678, arXiv:2502.00678 · DOI
  31. Cohen J (1992) A power primer. Psychol Bull 112(1):155–159. https://doi.org/10.1037/0033-2909.112.1.155 · doi:10.1037/0033-2909.112.1.155 · Paper page
  32. de Morais Leça M, Valença L, Santos R, de Souza Santos R (2025) Applications and implications of large language models in qualitative analysis: A new frontier for empirical software engineering. In: IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering, WSESE@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 36–43. https://doi.org/10.1109/WSESE66602.2025.00013 · DOI
  33. Dhar R, Vaidhyanathan K, Varma V (2024) Can LLMs generate architectural design decisions? - an exploratory empirical study. In: 21st IEEE International Conference on Software Architecture, ICSA 2024, Hyderabad, India, June 4-8, 2024, IEEE, pp 79–89. https://doi.org/10.1109/ICSA59870.2024.00016 · DOI
  34. Du X, Liu M, Wang K, Wang H, Liu J, Chen Y, Feng J, Sha C, Peng X, Lou Y (2024) Evaluating large language models in class-level code generation. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 81:1–81:13. https://doi.org/10.1145/3597503.3639219 · DOI · Paper page
  35. Dybå T, Kampenes VB, Sjøberg DIK (2006) A systematic review of statistical power in software engineering experiments. Inf Softw Technol 48(8):745–755. https://doi.org/10.1016/j.infsof.2005.08.009 · DOI · Paper page
  36. El-Hajjami A, Salinesi C (2025) How good are synthetic requirements? evaluating LLM-generated datasets for AI4RE. CoRR. https://doi.org/10.48550/ARXIV.2506.21138, arXiv:2506.21138 · arXiv · doi:10.48550/arxiv.2506.21138
  37. Evertz J, Risse N, Neuer N, Müller A, Normann P, Sapia G, Gupta S, Pape D, Shaw S, Srivastav D, Wressnegger C, Quiring E, Eisenhofer T, Arp D, Schönherr L (2026) Chasing shadows: Pitfalls in LLM security research. In: 33rd Annual Network and Distributed System Security Symposium, NDSS 2026, San Diego, California, USA, February 23-27, 2026, The Internet Society, https://www.ndss-symposium.org/ndss-paper/chasing-shadows-pitfalls-in-llm-security-research/ · DOI
  38. Gallegos IO, Rossi RA, Barrow J, Tanjim MM, Kim S, Dernoncourt F, Yu T, Zhang R, Ahmed NK (2024) Bias and fairness in large language models: A survey. Comput Linguist 50(3):1097–1179. https://doi.org/10.1162/COLI_A_00524 · DOI · Paper page
  39. Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, Demner-Fushman D, Dligach D, Daneshjou R, Fernandes C, Hansen LH, Landman A, Lehmann L, McCoy LG, Miller T, Moreno A, Munch N, Restrepo D, Savova G, Umeton R, Gichoya JW, Collins GS, Moons KGM, Celi LA, Bitterman DS (2025) The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med 31(1):60–69. https://doi.org/10.1038/s41591-024-03425-5 · DOI · Paper page
  40. Galster M, Mohsenimofidi S, Lulla JL, Abubakar MA, Treude C, Baltes S (2026) Configuring agentic AI coding tools: An exploratory study. In: Proceedings of the 3rd ACM international conference on AI-powered software (AIware 2026) · DOI
  41. Gerosa MA, Trinkenreich B, Steinmacher I, Sarma A (2024) Can AI serve as a substitute for human subjects in software engineering research? Autom Softw Eng 31(1):13. https://doi.org/10.1007/s10515-023-00409-6 · DOI
  42. Gibney E (2024) Not all ‘open source’ AI models are actually open. Nature News. https://doi.org/10.1038/d41586-024-02012-5 · DOI
  43. Golnari PA, Kumarappan A, Wen W, Liu X, Ryan G, Sun Y, Fu S, Nallipogu E (2026) DevBench: A realistic, developer-informed benchmark for code generation models. CoRR. https://doi.org/10.48550/ARXIV.2601.11895, arXiv:2601.11895 · DOI
  44. Gonçalves J, Silva M, Cabral B, Dias T, Maia E, Praça I, Severino R, Ferreira LL (2025) Evaluating LLaMA 3.2 for software vulnerability detection. In: Praça I, Bernardi S, Inácio PRM (eds) Cybersecurity - 9th European Interdisciplinary Cybersecurity Conference, EICC 2025, Rennes, France, June 18-19, 2025, Proceedings, Springer, Communications in Computer and Information Science, pp 38–51. https://doi.org/10.1007/978-3-031-94855-8_3 · DOI
  45. Guba EG (1981) Criteria for assessing the trustworthiness of naturalistic inquiries. ECTJ 29(2):75–91. https://doi.org/10.1007/BF02766777 · DOI · Paper page
  46. Guo D, Zhu Q, Yang D, Xie Z, Dong K, Zhang W, Chen G, Bi X, Wu Y, Li YK, Luo F, Xiong Y, Liang W (2024) DeepSeek-Coder: When the large language model meets programming - the rise of code intelligence. CoRR. https://doi.org/10.48550/ARXIV.2401.14196, arXiv:2401.14196 · DOI · Paper page
  47. Harding J, D’Alessandro W, Laskowski NG, Long R (2024) AI language models cannot replace human research participants. AI Soc 39(5):2603–2605. https://doi.org/10.1007/s00146-023-01725-x · DOI
  48. He J, Rungta M, Koleczek D, Sekhon A, Wang FX, Hasan S (2024a) Does prompt formatting have any impact on LLM performance? CoRR. https://doi.org/10.48550/ARXIV.2411.10541, arXiv:2411.10541 · DOI
  49. He Z, Huang C, Ding CC, Rohatgi S, Huang TK (2024b) If in a crowdsourced data annotation pipeline, a GPT-4. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI conference on human factors in computing systems, CHI 2024, ACM, pp 1040:1–1040:25. https://doi.org/10.1145/3613904.3642834 · DOI
  50. Holtzman A, Buys J, Du L, Forbes M, Choi Y (2020) The curious case of neural text degeneration. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net. https://openreview.net/forum?id=rygGQyrFvH
  51. Hosking T, Blunsom P, Bartolo M (2024) Human feedback is not gold standard. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=7W3GLNImfS
  52. Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, Luo X, Lo D, Grundy J, Wang H (2024) Large language models for software engineering: A systematic literature review. ACM Trans Softw Eng Methodol 33(8):220:1-220:79. https://doi.org/10.1145/3695988 · DOI · Paper page
  53. Hu X, Niu F, Chen J, Zhou X, Zhang J, He J, Xia X, Lo D (2025) Assessing and advancing benchmarks for evaluating large language models in software engineering tasks. CoRR. https://doi.org/10.48550/ARXIV.2505.08903, arXiv:2505.08903 · arXiv · doi:10.48550/arxiv.2505.08903
  54. Huang F, Kwak H, An J (2023) Is ChatGPT better than human annotators? potential and limitations of ChatGPT in explaining implicit hate speech. In: Ding Y, Tang J, Sequeda JF, Aroyo L, Castillo C, Houben G (eds) Companion Proceedings of the ACM Web Conference 2023, WWW 2023, ACM, pp 294–297. https://doi.org/10.1145/3543873.3587368 · DOI · Paper page
  55. Huang J, Yi B, Sun W, Wan B, Xu Y, Feng Y, Ye W, Qin Q (2024) Enhancing review classification via LLM-based data annotation and multi-perspective feature representation learning. SSRN Electronic Journal pp 1–15. https://doi.org/10.2139/ssrn.5002351, https://ssrn.com/abstract=5002351 · DOI
  56. Hui B, Yang J, Cui Z, Yang J, Liu D, Zhang L, Liu T, Zhang J, Yu B, Dang K, Yang A, Men R, Huang F, Ren X, Ren X, Zhou J, Lin J (2024) Qwen2.5-Coder technical report. CoRR. https://doi.org/10.48550/ARXIV.2409.12186, arXiv:2409.12186 · DOI · Paper page
  57. Husain H, Wu H, Gazit T, Allamanis M, Brockschmidt M (2019) CodeSearchNet challenge: Evaluating the state of semantic code search. CoRR. arXiv:1909.09436 · arXiv
  58. Hymel C, Johnson H (2025) Analysis of LLMs vs human experts in requirements engineering. CoRR. https://doi.org/10.48550/ARXIV.2501.19297, arXiv:2501.19297 · DOI
  59. Jahic J, Sami A (2024) State of practice: LLMs in software engineering and software architecture. In: 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyderabad, India, June 4-8, 2024, IEEE, pp 311–318. https://doi.org/10.1109/ICSA-C63560.2024.00059 · DOI
  60. Jain N, Han K, Gu A, Li W, Yan F, Zhang T, Wang S, Solar-Lezama A, Sen K, Stoica I (2024) LiveCodeBench: Holistic and contamination free evaluation of large language models for code. CoRR. https://doi.org/10.48550/ARXIV.2403.07974, arXiv:2403.07974 · DOI · Paper page
  61. Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR (2024) SWE-bench: Can language models resolve real-world github issues? In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66
  62. Jowsey T, Braun V, Clarke V, Lupton D, Fine M (2025) We reject the use of generative artificial intelligence for reflexive qualitative research. Qual Inq. https://doi.org/10.1177/10778004251401851 · DOI
  63. Kang S, Yoon J, Yoo S (2023) Large language models are few-shot testers: Exploring LLM-based general bug reproduction. In: 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, IEEE, pp 2312–2323. https://doi.org/10.1109/ICSE48619.2023.00194 · DOI · Paper page
  64. Kapoor S, Cantrell EM, Peng K, Pham TH, Bail CA, Gundersen OE, Hofman JM, Hullman J, Lones MA, Malik MM, Nanayakkara P, Poldrack RA, Raji ID, Roberts M, Salganik MJ, Serra-Garcia M, Stewart BM, Vandewiele G, Narayanan A (2024) REFORMS: Consensus-based recommendations for machine-learning-based science. Sci Adv 10(18):eadk3452. https://doi.org/10.1126/sciadv.adk3452 · DOI
  65. Khojah R, Mohamad M, Leitner P, de Oliveira Neto FG (2024) Beyond code generation: An observational study of ChatGPT usage in software engineering practice. Proc ACM Softw Eng 1(FSE):1819–1840. https://doi.org/10.1145/3660788 · DOI
  66. Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K (2024) Can large language models replace humans in systematic reviews? evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods 15(4):616–626. https://doi.org/10.1002/jrsm.1715, https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1715 · DOI · Paper page
  67. Kirk R, Mediratta I, Nalmpantis C, Luketina J, Hambro E, Grefenstette E, Raileanu R (2023) Understanding the effects of RLHF on LLM generalisation and diversity. CoRR. arXiv:2310.06452 · arXiv
  68. Kong A, Zhao S, Chen H, Li Q, Qin Y, Sun R, Zhou X, Wang E, Dong X (2024) Better zero-shot reasoning with role-play prompting. In: Duh K, Gómez-Adorno H, Bethard S (eds) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, Association for Computational Linguistics, pp 4099–4113. https://doi.org/10.18653/V1/2024.NAACL-LONG.228 · DOI
  69. Korn A, Zaruchas L, Arora C, Metzger A, Smolka S, Wang F, Vogelsang A (2026) Reporting LLM prompting in automated software engineering: A guideline based on current practices and expectations. CoRR. https://doi.org/10.48550/ARXIV.2601.01954, to appear at FORGE 2026, co-located with ICSE 2026, Rio de Janeiro, Brazil. arXiv:2601.01954 · arXiv · doi:10.48550/arxiv.2601.01954
  70. Krippendorff K (2018) Content analysis: An introduction to its methodology, 4th edn. SAGE Publications · DOI · Paper page
  71. Kübler JM, Budhathoki K, Kleindessner M, Zhou X, Yin J, Khetan A, Karypis G (2026) When LLMs get significantly worse: A statistical approach to detect model degradations. In: Proceedings of the International Conference on Learning Representations (ICLR)
  72. Kulal S, Pasupat P, Chandra K, Lee M, Padon O, Aiken A, Liang P (2019) SPoC: Search-based pseudocode to code. In: Wallach HM, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox EB, Garnett R (eds) Advances in neural information processing systems 32: Annual conference on neural information processing systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp 11883–11894. https://proceedings.neurips.cc/paper/2019/hash/7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html
  73. Lago P, Runeson P, Song Q, Verdecchia R (2024) Threats to validity in software engineering - hypocritical paper section or essential analysis? In: Franch X, Daneva M, Martínez-Fernández S, Quaranta L (eds) Proceedings of the 18th ACM/IEEE international symposium on Empirical Software Engineering and Measurement, ESEM 2024, Barcelona, Spain, October 24-25, 2024, ACM, pp 314–324. https://doi.org/10.1145/3674805.3686691 · DOI
  74. Li D, Gupta K, Bhaduri M, Sathiadoss P, Bhatnagar S, Chong J (2024) Comparing GPT-3.5 and GPT-4 accuracy and drift in radiology diagnosis please cases. Radiology 310(1):e232411. https://doi.org/10.1148/radiol.232411 · DOI
  75. Li R, Allal LB, Zi Y, Muennighoff N, Kocetkov D, Mou C, Marone M, Akiki C, Li J, Chim J, Liu Q, Zheltonozhskii E, Zhuo TY, Wang T, Dehaene O, Davaadorj M, Lamy-Poirier J, Monteiro J, Shliazhko O, Gontier N, Meade N, Zebaze A, Yee M, Umapathi LK, Zhu J, Lipkin B, Oblokulov M, Wang Z, V RM, Stillerman JT, Patel SS, Abulkhanov D, Zocca M, Dey M, Zhang Z, Fahmy N, Bhattacharyya U, Yu W, Singh S, Luccioni S, Villegas P, Kunakov M, Zhdanov F, Romero M, Lee T, Timor N, Ding J, Schlesinger C, Schoelkopf H, Ebert J, Dao T, Mishra M, Gu A, Robinson J, Anderson CJ, Dolan-Gavitt B, Contractor D, Reddy S, Fried D, Bahdanau D, Jernite Y, Ferrandis CM, Hughes S, Wolf T, Guha A, von Werra L, de Vries H (2023) StarCoder: may the source be with you! Trans Mach Learn Res 2023. https://openreview.net/forum?id=KoFOg41haE
  76. Liang JT, Badea C, Bird C, DeLine R, Ford D, Forsgren N, Zimmermann T (2024) Can GPT-4 replicate empirical software engineering research? Proc ACM Softw Eng 1(FSE):1330–1353. https://doi.org/10.1145/3660767 · DOI · Paper page
  77. Liu J, Xia CS, Wang Y, Zhang L (2023) Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In: Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S (eds) Advances in neural information processing systems 36: Annual conference on neural information processing systems 2023, NeurIPS 2023, http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
  78. Liu M, Huang X, He W, Xie Y, Zhang JM, Jing X, Chen Z, Ma Y (2024a) Research artifacts in software engineering publications: Status and trends. J Syst Softw 213:112032. https://doi.org/10.1016/j.jss.2024.112032 · DOI · Paper page
  79. Liu X, Yu H, Zhang H, Xu Y, Lei X, Lai H, Gu Y, Ding H, Men K, Yang K, Zhang S, Deng X, Zeng A, Du Z, Zhang C, Shen S, Zhang T, Su Y, Sun H, Huang M, Dong Y, Tang J (2024b) AgentBench: Evaluating LLMs as agents. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=zAdUB0aCTQ
  80. Lubos S, Felfernig A, Tran TNT, Garber D, Mansi ME, Erdeniz SP, Le V (2024) Leveraging LLMs for the quality assurance of software requirements. In: Liebel G, Hadar I, Spoletini P (eds) 32nd IEEE International Requirements Engineering Conference, RE 2024, Reykjavik, Iceland, June 24-28, 2024, IEEE, pp 389–397. https://doi.org/10.1109/RE59067.2024.00046 · DOI
  81. Madampe K, Grundy J, Hoda R, Obie HO (2024) The struggle is real! the agony of recruiting participants for empirical software engineering studies. In: 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Liverpool, UK, September 2-6, 2024, IEEE, pp 417–422. https://doi.org/10.1109/VL/HCC60511.2024.00065 · DOI
  82. Martinez Montes C, Feldt R, Martos CM, Ouhbi S, Premanandan S, Graziotin D (2025) Large language models in thematic analysis: Prompt engineering, evaluation, and guidelines for qualitative software engineering research. CoRR. https://doi.org/10.48550/ARXIV.2510.18456, arXiv:2510.18456 · arXiv · doi:10.48550/arxiv.2510.18456
  83. Menzies T (2025) The case for compact AI. Commun ACM 68(9):6–7. https://doi.org/10.1145/3746057 · doi:10.1145/3746057
  84. Mitu NE, Mitu GT (2024) The hidden cost of AI: Carbon footprint and mitigation strategies. Revista de Stiinte Politice Revue des Sci Politiq 84:9–16. https://doi.org/10.2139/ssrn.5036344 · doi:10.2139/ssrn.5036344
  85. Mohamed S, Parvin A, Parra E (2024) Chatting with AI: deciphering developer conversations with ChatGPT. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International conference on mining software repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 187–191. https://doi.org/10.1145/3643991.3645078 · DOI
  86. Moumoula MB, Kaboré AK, Klein J, Bissyandé TF (2024) Large language models for cross-language code clone detection. CoRR. https://doi.org/10.48550/ARXIV.2408.04430, arXiv:2408.04430 · arXiv · doi:10.48550/arxiv.2408.04430
  87. Navarro KF, Syriani E, Arawjo I (2026) Reporting and reviewing LLM-integrated systems in HCI: challenges and considerations. In: Oliver N, Shamma DA, Candello H, César P, Lopes P, Bozzon A, Kosch T, Liao VQ, Ma X, Artizzu V, Draxler F, López G, Reinschluessel AV, Tong X, Dugas POT (eds) Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026, Barcelona, Spain, April 13-17, 2026, ACM, pp 1546:1–1546:18. https://doi.org/10.1145/3772318.3790439 · DOI
  88. Ornelas T, Araújo AA, Araújo J, Araújo M, Trinkenreich B, Kalinowski M (2025) LLM-assisted thematic analysis: Opportunities, limitations, and recommendations. CoRR. https://doi.org/10.48550/ARXIV.2511.14528 , arXiv:2511.14528 · arXiv · doi:10.48550/arxiv.2511.14528
  89. Pangakis N, Wolken S, Fasching N (2023) Automated annotation with generative AI requires validation. CoRR. https://doi.org/10.48550/ARXIV.2306.00176, arXiv:2306.00176 · DOI · Paper page
  90. Panickssery A, Bowman SR, Feng S (2024) LLM evaluators recognize and favor their own generations. In: Globersons A, Mackey L, Belgrave D, Fan A, Paquet U, Tomczak JM, Zhang C (eds) Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. http://papers.nips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html
  91. Papineni K, Roukos S, Ward T, Zhu W (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, pp 311–318, https://doi.org/10.3115/1073083.1073135, https://aclanthology.org/P02-1040/ · DOI · Paper page
  92. Pezeshkpour P, Hruschka E (2024) Large language models sensitivity to the order of options in multiple-choice questions. In: Duh K, Gómez-Adorno H, Bethard S (eds) Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, Association for Computational Linguistics, Findings of ACL, vol NAACL 2024, pp 2006–2017. https://doi.org/10.18653/V1/2024.FINDINGS-NAACL.130 · DOI · Paper page
  93. Pineau J, Vincent-Lamarre P, Sinha K, Larivière V, Beygelzimer A, d’Alché-Buc F, Fox EB, Larochelle H (2021) Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). J Mach Learn Res 22:164:1–164:20. https://jmlr.org/papers/v22/20-303.html
  94. Rabbi MF, Champa AI, Zibran MF, Islam MR (2024) AI writes, we analyze: The ChatGPT Python code saga. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 177–181. https://doi.org/10.1145/3643991.3645076 · DOI
  95. Ralph P, Tempero ED (2018) Construct validity in software engineering research and software metrics. In: Rainer A, MacDonell SG, Keung JW (eds) Proceedings of the 22nd international conference on Evaluation and Assessment in Software Engineering, EASE2018, ACM, pp 13–23. https://doi.org/10.1145/3210459.3210461 · DOI · Paper page
  96. Ralph P, bin Ali N, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A, de França BBN, Furia CA, Gay G, Gold N, Graziotin D, He P, Hoda R, Juristo N, Kitchenham B, Lenarduzzi V, Martínez J, Melegati J, Mendez D, Menzies T, Molleri J, Pfahl D, Robbes R, Russo D, Saarimäki N, Sarro F, Taibi D, Siegmund J, Spinellis D, Staron M, Stol K, Storey MA, Taibi D, Tamburri D, Torchiano M, Treude C, Turhan B, Wang X, Vegas S (2021) Empirical standards for software engineering research. arXiv:2010.03525 · arXiv
  97. Ralph P, Kuutila M, Arif H, Ayoola B (2024) Teaching software metrology: The science of measurement for software engineering. In: Méndez D, Avgeriou P, Kalinowski M, Ali NB (eds) Handbook on teaching empirical software engineering, Springer Nature Switzerland, pp 101–154. https://doi.org/10.1007/978-3-031-71769-7_5 · DOI
  98. Rasheed Z, Waseem M, Ahmad A, Kemell K, Wang X, Nguyen-Duc A, Abrahamsson P (2024) Can large language models serve as data analysts? A multi-agent assisted approach for qualitative data analysis. CoRR. https://doi.org/10.48550/ARXIV.2402.01386, arXiv:2402.01386 · DOI
  99. Reiss MV (2023) Testing the reliability of ChatGPT for text annotation and classification: A cautionary remark. CoRR. https://doi.org/10.48550/ARXIV.2304.11085, arXiv:2304.11085 · DOI · Paper page
  100. Ren S, Guo D, Lu S, Zhou L, Liu S, Tang D, Sundaresan N, Zhou M, Blanco A, Ma S (2020) CodeBLEU: a method for automatic evaluation of code synthesis. CoRR arXiv:2009.10297 · arXiv
  101. Renze M (2024) The effect of sampling temperature on problem solving in large language models. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp 7346–7356. https://doi.org/10.18653/v1/2024.findings-emnlp.432 · DOI · Paper page
  102. Richards J, Wessel M (2024) What you need is what you get: Theory of mind for an LLM-based code understanding assistant. In: IEEE International Conference on Software Maintenance and Evolution, ICSME 2024, IEEE, pp 666–671, https://doi.org/10.1109/ICSME58944.2024.00070 · DOI
  103. Robillard MP, Arya DM, Ernst NA, Guo JLC, Lamothe M, Nassif M, Novielli N, Serebrenik A, Steinmacher I, Stol K (2024) Communicating study design trade-offs in software engineering. ACM Trans Softw Eng Methodol 33(5):112:1-112:10. https://doi.org/10.1145/3649598 · DOI
  104. Ronanki K, Berger C, Horkoff J (2023) Investigating ChatGPT’s potential to assist in requirements elicitation processes. In: 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, pp 354–361 · DOI
  105. Rozière B, Lachaux M, Chanussot L, Lample G (2020) Unsupervised translation of programming languages. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H (eds) Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. https://proceedings.neurips.cc/paper/2020/hash/ed23fbf18c2cd35f8c7f8de44f85c08d-Abstract.html
  106. Rozière B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, Adi Y, Liu J, Remez T, Rapin J, Kozhevnikov A, Evtimov I, Bitton J, Bhatt M, Canton-Ferrer C, Grattafiori A, Xiong W, Défossez A, Copet J, Azhar F, Touvron H, Martin L, Usunier N, Scialom T, Synnaeve G (2023) Code Llama: Open foundation models for code. CoRR. https://doi.org/10.48550/ARXIV.2308.12950, arXiv:2308.12950 · DOI · Paper page
  107. Russo D (2024) Navigating the complexity of generative AI adoption in software engineering. ACM Trans Softw Eng Methodol 33(5):135:1-135:50. https://doi.org/10.1145/3652154 · DOI · Paper page
  108. Sallou J, Durieux T, Panichella A (2024) Breaking the silence: the threats of using LLMs in software engineering. In: Proceedings of the 2024 ACM/IEEE 44th international conference on software engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, pp 102–106. https://doi.org/10.1145/3639476.3639764 · DOI
  109. Santos A, Vegas S, Oivo M, Juristo N (2021) A procedure and guidelines for analyzing groups of software engineering replications. IEEE Trans Software Eng 47(9):1742–1763. https://doi.org/10.1109/TSE.2019.2935720 · DOI
  110. Schäfer M, Nadi S, Eghbali A, Tip F (2024) An empirical evaluation of using large language models for automated unit test generation. IEEE Trans Software Eng 50(1):85–105. https://doi.org/10.1109/TSE.2023.3334955 · DOI · Paper page
  111. Schilke O, Reimann M (2025) The transparency dilemma: How AI disclosure erodes trust. Organ Behav Hum Decis Process 188:104405. https://doi.org/10.1016/j.obhdp.2025.104405 , https://www.sciencedirect.com/science/article/pii/S0749597825000172 · DOI
  112. Schneider K, Fotrousi F, Wohlrab R (2025) A reference model for empirically comparing LLMs with humans. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Society, ICSE-SEIS 2025, Ottawa, ON, Canada, April 27 - May 3, 2025. IEEE, pp 130–134. https://doi.org/10.1109/ICSE-SEIS66351.2025.00018 · DOI
  113. Schröder S, Morgenroth T, Kuhl U, Vaquet V, Paaßen B (2025) Large language models do not simulate human psychology. CoRR. https://doi.org/10.48550/ARXIV.2508.06950, arXiv:2508.06950 · DOI
  114. Schroeder K, Wood-Doughty Z (2024) Can you trust LLM judgments? reliability of LLM-as-a-judge. CoRR. https://doi.org/10.48550/ARXIV.2412.12509, arXiv:2412.12509 · DOI · Paper page
  115. Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332. https://doi.org/10.1136/bmj.c332 · DOI · Paper page
  116. Sclar M, Choi Y, Tsvetkov Y, Suhr A (2024) Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net. https://openreview.net/forum?id=RIu5lyNXjT
  117. Shull F, Singer J, Sjøberg DIK (eds) (2008) Guide to advanced empirical software engineering. Springer. https://doi.org/10.1007/978-1-84800-044-5 · DOI · Paper page
  118. Silva A, Monperrus M (2025) RepairBench: Leaderboard of frontier models for program repair. In: IEEE/ACM International workshop on large language models for code, LLM4Code@ICSE 2025, Ottawa, ON, Canada, May 3, 2025, IEEE, pp 9–16. https://doi.org/10.1109/LLM4CODE66737.2025.00006 · DOI
  119. Sjøberg DIK, Bergersen GR (2023) Construct validity in software engineering. IEEE Trans Software Eng 49(3):1374–1396. https://doi.org/10.1109/TSE.2022.3176725 · DOI
  120. Song Y, Wang G, Li S, Lin BY (2025) The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. In: Chiruzzo L, Ritter A, Wang L (eds) Proceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: Human language technologies, NAACL 2025 - Volume 1: Long Papers, Association for Computational Linguistics, pp 4195–4206. https://doi.org/10.18653/V1/2025.NAACL-LONG.211 · DOI
  121. Sonnekalb T, Gruner B, Brust C, Mäder P (2022) Generalizability of code clone detection on CodeBERT. In: 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, ACM, pp 143:1–143:3. https://doi.org/10.1145/3551349.3561165 · DOI
  122. Sullivan GM, Sargeant J (2011) Qualities of qualitative research: part I. J Grad Med Educ 3(4):449–452 · DOI
  123. Sumers TR, Yao S, Narasimhan K, Griffiths TL (2024) Cognitive architectures for language agents. Trans Mach Learn Res 2024. https://openreview.net/forum?id=1i6ZCvflQJ
  124. Takerngsaksiri W, Pasuksmit J, Thongtanunam P, Tantithamthavorn C, Zhang R, Jiang F, Li J, Cook E, Chen K, Wu M (2025) Human-in-the-loop software development agents. In: 47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025, IEEE, pp 342–352. https://doi.org/10.1109/ICSE-SEIP66354.2025.00036 · DOI · Paper page
  125. Verdecchia R, Engström E, Lago P, Runeson P, Song Q (2023) Threats to validity in software engineering research: A critical reflection. Inf Softw Technol 164:107329. https://doi.org/10.1016/j.infsof.2023.107329 · DOI
  126. von Kistowski J, Arnold JA, Huppler K, Lange K, Henning JL, Cao P (2015) How to build a benchmark. In: John LK, Smith CU, Sachs K, Lladó CM (eds) Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, Austin, TX, USA, January 31 - February 4, 2015, ACM, pp 333–336. https://doi.org/10.1145/2668930.2688819 · DOI
  127. Wagner S, Muñoz Barón M, Falessi D, Baltes S (2025) Towards evaluation guidelines for empirical studies involving LLMs. In: IEEE/ACM International workshop on methodological issues with empirical studies in software engineering, WSESE@ICSE 2025, May 3, 2025, IEEE, pp 24–27. https://doi.org/10.1109/WSESE66602.2025.00011 · DOI
  128. Wang S, Liu Y, Xu Y, Zhu C, Zeng M (2021) Want to reduce labeling cost? GPT-3 can help. In: Moens M, Huang X, Specia L, Yih SW (eds) Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, Association for Computational Linguistics, Findings of ACL, vol EMNLP 2021, pp 4195–4205. https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.354 · DOI · Paper page
  129. Wang C, Huang K, Zhang J, Feng Y, Zhang L, Liu Y, Peng X (2024a) How and why LLMs use deprecated APIs in code completion? an empirical study. CoRR. https://doi.org/10.48550/ARXIV.2406.09834 , arXiv:2406.09834 · DOI
  130. Wang X, Kim H, Rahman S, Mitra K, Miao Z (2024b) Human-LLM collaborative annotation through effective verification of LLM labels. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI conference on human factors in computing systems, CHI 2024, ACM, pp 303:1–303:21. https://doi.org/10.1145/3613904.3641960 · DOI
  131. Wang A, Morgenstern J, Dickerson JP (2025a) Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat Mac Intell 7(3):400–411. https://doi.org/10.1038/s42256-025-00986-z · DOI
  132. Wang F, Arora C, Liu Y, Huang K, Tantithamthavorn C, Aleti A, Sambathkumar D, Lo D (2025b) Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs. In: 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, pp 3334–3345. https://doi.org/10.1109/ASE63991.2025.00275 · DOI
  133. Widder DG, Whittaker M, West SM (2024) Why ‘open’AI systems are actually closed, and why this matters. Nature 635(8040):827–833 · DOI · Paper page
  134. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2024) Experimentation in software engineering, 2nd edn. Springer. https://doi.org/10.1007/978-3-662-69306-3 · DOI · Paper page
  135. Xia Y, Shao H, Deng X (2024) VulCoBERT: A CodeBERT-based system for source code vulnerability detection. In: 2024 International conference on Generative Artificial Intelligence and Information Security, GAIIS 2024, Kuala Lumpur, Malaysia, May 10-12, 2024, ACM. https://doi.org/10.1145/3665348.3665391 · DOI
  136. Xiao T, Treude C, Hata H, Matsumoto K (2024) DevGPT: Studying developer-ChatGPT conversations. In: Spinellis D, Bacchelli A, Constantinou E (eds) 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, pp 227–230. https://doi.org/10.1145/3643991.3648400 · DOI · Paper page
  137. Xu C, Guan S, Greene D, Kechadi MT (2024a) Benchmark data contamination of large language models: A survey. CoRR. https://doi.org/10.48550/ARXIV.2406.04244, arXiv:2406.04244 · DOI · Paper page
  138. Xu R, Sun Y, Ren M, Guo S, Pan R, Lin H, Sun L, Han X (2024b) AI for social science and social science of AI: A survey. Inf Process Manag 61(2):103665. https://doi.org/10.1016/J.IPM.2024.103665 · DOI
  139. Yan L, Hwang A, Wu Z, Head A (2024) Ivie: Lightweight anchored explanations of just-generated code. In: Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas POT, Shklovski I (eds) Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, ACM, pp 140:1–140:15. https://doi.org/10.1145/3613904.3642239 · DOI
  140. Yang G, Zhou Y, Chen X, Zhang X, Han T, Chen T (2023) ExploitGen: Template-augmented exploit code generation based on CodeBERT. J Syst Softw 197:111577. https://doi.org/10.1016/J.JSS.2022.111577 · DOI
  141. Yang Y, Shin J, Choi B, Park M, Ju D, Lee C, Chun S, Kang D, Kim J, Yoon S (2025) From code foundation models to agents and applications: A comprehensive survey and practical guide to code intelligence. arXiv preprint arXiv:2502.11827 · arXiv
  142. Zhang J, Yu S, Chong D, Sicilia A, Tomz MR, Manning CD, Shi W (2025) Verbalized sampling: How to mitigate mode collapse and unlock LLM diversity. CoRR. arXiv:2510.01171 · arXiv
  143. Zhao C, Habule M, Zhang W (2025) Large language models (LLMs) as research subjects: Status, opportunities and challenges. New Ideas Psychol 79:101167. https://doi.org/10.1016/j.newideapsych.2025.101167, https://www.sciencedirect.com/science/article/pii/S0732118X25000236 · DOI
  144. Zhou R, Chen L, Yu K (2024) Is LLM a reliable reviewer? A comprehensive evaluation of LLM on automatic paper reviewing tasks. In: Calzolari N, Kan M, Hoste V, Lenci A, Sakti S, Xue N (eds) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, ELRA and ICCL, pp 9340–9351. https://aclanthology.org/2024.lrec-main.816 · DOI · Paper page
  145. Zhu Y, Zhang P, ul Haq E, Hui P, Tyson G (2023) Can ChatGPT reproduce human-generated labels? A study of social computing tasks. CoRR. https://doi.org/10.48550/ARXIV.2304.10145, arXiv:2304.10145 · DOI

Metadata

Authors
Baltes, S., Angermeir, F., Arora, C., Muñoz Barón, M., Chen, C., Böhme, L., Calefato, F., Ernst, N., Falessi, D., Fitzgerald, B., Fucci, D., He, J., Treude, C., Kalinowski, M., Lambiase, S., Russo, D., Lungu, M., Martinez Montes, C., Prechelt, L., Ralph, P., van Tonder, R., & Wagner, S.
Venue
Empirical Software Engineering
Publisher
Springer Science and Business Media LLC
Published
2026-11-03
Volume
32
Issue
2
ISSN
  • 1382-3256
  • 1573-7616
DOI
10.1007/s10664-026-10922-3
Cited by
3
References
16
Language
en
Licence
Funders
Medizinische Fakultät Heidelberg der Universität Heidelberg

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Guidelines for empirical studies in software engineering involving large language models. https://app.cochat.ai/papers/10.1007/s10664-026-10922-3
The paper
Baltes, S., Angermeir, F., Arora, C., Muñoz Barón, M., Chen, C., Böhme, L., Calefato, F., Ernst, N., Falessi, D., Fitzgerald, B., Fucci, D., He, J., Treude, C., Kalinowski, M., Lambiase, S., Russo, D., Lungu, M., Martinez Montes, C., Prechelt, L., . . . Wagner, S. (2027). Guidelines for empirical studies in software engineering involving large language models. Empirical Software Engineering. https://doi.org/10.1007/s10664-026-10922-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.