An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 6 of the 45 sources its claims cite were checked for support.
Abstract
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.
Overview
CoChat Summary
TL;DR
This paper empirically studies non-determinism in ChatGPT code generation across 829 problems from CodeContests, APPS, and HumanEval. It finds substantial semantic, syntactic, and structural variation across repeated identical prompts, and shows that temperature 0 does not guarantee determinism.
Key findings
- Under default settings, the ratio of coding tasks with zero equal test output among five requests was 75.76% for CodeContests, 51.00% for APPS, and 47.56% for HumanEval; maximum test-pass-rate difference reached 1.00 on all three datasets and accounted for 39.63% of HumanEval problems.
- Setting temperature to 0 reduced but did not eliminate non-determinism: there were still 43.64% (CodeContests), 27.40% (APPS), and 18.29% (HumanEval) of problems with no equal test output among five candidates.
- GPT-4 was slightly more deterministic than GPT-3.5 at temperature 1, but their non-determinism was similar at temperature 0; Chain-of-Thought prompting led to more non-determinism at temperature 0, while prompts requesting concise code were more stable.
- In a manual review of 76 LLM-based code generation papers from the last two years, only 21.1% (16/76) considered non-determinism in their experimental evaluation.
Methods
- Empirical study of 829 code generation problems: CodeContests (165), APPS (500 randomly sampled), and HumanEval (164), using ChatGPT API models `gpt-3.5-turbo-0125` and `gpt-4-0613` with zero-shot Python3 prompts.
- For each task, the API was called five times with the same prompt; generated code was extracted, compiled, and run against benchmark test suites with a 3-second timeout.
- Non-determinism was measured via semantic similarity (test pass rate, output equivalence rate), syntactic similarity (Longest Common Subsequence, Levenshtein Edit Distance), and structural similarity (AST-based `pycode_similar` with Unified_Diff and Tree_Diff), with Kruskal-Wallis tests for significance.
Limitations
- The paper states that it does not study different programming languages, uses only a few code similarity measures, does not fully consider prompt influence on non-determinism, and focuses exclusively on ChatGPT rather than other LLMs.
- It also states that measuring semantic similarity through test execution outputs may oversimplify semantic similarity by ignoring code meaning, logic, and diverse correct solutions.
- The text makes evident that several analyses (RQ2, RQ3, RQ5, RQ6) report main results only for CodeContests, with other dataset results placed on the homepage, limiting the in-paper generalizability of those comparisons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 2 Not valid · 18 Not checked · 125 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
What the sources say
- Partly supportedcites [15] [15] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. arXiv:2310.03533 [cs.SE] https: //arxiv.org/abs/2310.03533 - sourceExperimental — accuracy not yet measured
Our copy treats ChatGPT as a dominant subject of code-generation research — the background section ("A Large Language Model (LLM) refers to...", p. n/a) says the step change in GPT-3/3.5 performance attracted "a great deal of interest in GPT (and ChatGPT) in particular", and Section D (Scientific Evaluation of LLM-based Code Generation) surveys ChatGPT-centred work (Borji's failure analysis, Ouyang et al.'s non-determinism study, Yetistiren et al.'s comparison). However, the paper never states the superlative the claim attributes to it: it reports no counts or ranking of how often each LLM is used across the code-generation literature, and the only adoption figure it gives (Section C, "46% of the developers' code was written by Copilot", i.e. Codex/GPT-4-based Copilot in industry) is about practitioner use, not literature uptake. So the "widely used/much-studied" part is supported while "the most widely adopted" is not established by this source. (LLM, conf=0.68)
- Not checked yetcites [16] [16] Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE Computer Society, Los Alamitos, CA, USA, 876-885. https://doi.org/10.1109/COMPSAC57700.2023.00117 - source
- Partly supportedcites [23] [23] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2023. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. arXiv:2309.08221 [cs.SE] https: //arxiv.org/abs/2309.08221 - sourceExperimental — accuracy not yet measured
Section 2.2 identifies ChatGPT as an LLM and describes this paper's use of it for code refinement. Related Work says ChatGPT has been evaluated on several software-engineering tasks. Neither passage establishes that it is the most widely adopted LLM in the code-generation literature. (LLM, conf=0.83)
- Partly supportedcites [40] [40] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 - sourceExperimental — accuracy not yet measured
The Additional information section includes ChatGPT among 26 models described as representative and popular for evaluation, and the paper evaluates its code generation. It does not compare adoption across the literature or establish that ChatGPT is the most widely adopted LLM for code generation. (LLM, conf=0.84)
- Partly supportedcites [42] [42] Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1-26. - sourceExperimental — accuracy not yet measured
Section 8.1 says ChatGPT has gained popularity for code generation, and the Introduction describes its rapid growth in users. Neither passage establishes that it is the most widely adopted LLM for code generation in the literature; the cited paper provides no comparison of adoption across models or studies. (LLM, conf=0.85)
- Partly supportedcites [63] [63] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Trans. Softw. Eng. 50, 4 (Feb. 2024), 911-936. https: //doi.org/10.1109/TSE.2024.3368208 - sourceExperimental — accuracy not yet measured
Section 5.1 LLM Models states that 'the most commonly utilized LLM in software testing tasks is ChatGPT' and that it 'ranks in the top position in terms of our collected studies' (102 papers), so the source does support ChatGPT being the most adopted LLM in the software-testing literature it surveyed. However, the claim's qualifier 'in code generation' is not what the source reports: the Introduction explicitly separates the two domains ('software testing possesses unique characteristics that differentiate it from code generation'), and Section 7 credits the NL2Code/code-generation survey to Zan et al. [168], a different work. The ranking is also bounded to the authors' collected studies rather than the whole literature. (LLM, conf=0.78)
- Not checked yetcites [70] [70] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24) . Association for Computing Machinery, New York, NY, USA, Article 37, 12 pages. https://doi.org/10.1145/3597503.3623316 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, non-determinism affects the reliability and reproducibility of empirical software engineering [53].”
What the sources say
- Could not verifycites [53] [53] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv:2201.11227 [cs.LG] https: //arxiv.org/abs/2201.11227 - sourceExperimental — accuracy not yet measured
Our copy of Synchromesh (Poesia et al., 2022) is about the reliability of LLM code generation — i.e., preventing syntactic/semantic implementation errors via Target Similarity Tuning and Constrained Semantic Decoding (Abstract p.1; Intro p.1-2; Discussion p.8-9) — not about non-determinism undermining empirical software engineering. The nearest content is its use of stochastic sampling (temperature τ = 0.7, Valid@K, Results p.8) and the observation that LLMs are sensitive to which few-shot examples are in the prompt (Intro p.2), but no passage frames non-determinism as a threat to reliability or reproducibility of empirical software engineering. Searched: 'non-determinism / reproducibility / empirical software engineering', 'stochastic sampling temperature reproducibility of experiments', 'deterministic decoding / outputs vary across runs', 'validity of experiments'; the copy is not truncated, so the point is simply absent rather than missing. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for three different code generation datasets (i.e., CodeContests, APPS, and HumanEval), respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we collect 76 LLM-based code generation papers that appeared in the last 2 years. Our manual analysis results highlight that only 21.1% of these papers consider the non-determinism threat in their experiments.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“accounts for 39.63% of the problems in HumanEval, the most widely used code generation benchmark.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
80 more claims
Create a free account to see all 146 claims and to verify the 38 references still open.
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
- Probably not a substantial claim. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One major issue is the inherent stochasticity in the models’ architecture, particularly in transformer-based models, which utilize probabilistic techniques during inference (e.g., beam search or temperature sampling)125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, Ouyang et al. reported that only a minority of studies take non-determinism into account in their experimental evaluation when using ChatGPT for code generation, suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent work has identified multiple sources of non-determinism in large language models.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
The cited source says what the claim attributes to it.
In the citing paper
“For code generation specifically, Ouyang et al. (2025) find that repeated queries yield different implementations even with greedy sampling.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.2 RQ2: Influence of Temperature · p. 13
“When temperature =0.5, we observe that ChatGPT tends to generate code candidates that are more deterministic than temperature=1, but less deterministic than temperature=0.”
§ 4.2 RQ2: Influence of Temperature · p. 13
“| 0 | 0.41 | 0.39 | 0.67 | 0.50 | 0.46 | 0.74 | | 0.5 | 0.61 | 0.49 | 0.63 | 0.69 | 0.58 | 0.71 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 1 Introduction · p. 2
“the ratio of problems with not a single equal test output among the top-five code candidates is above 50% for all the benchmarks we study”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the ratios of tasks with zero equal test output (i.e., OER=0) among the five code candidates are 75.76%, 51.00%, and 47.56% for the three datasets, respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“For temperature=1, we can observe that GPT-4 is slightly more deterministic than GPT-3.5, with lower test pass rate variance, lower ratio of worst cases, lower OER and OER (no ex.), lower LCS, higher LED, and lower structural similarity under two settings.”
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“| GPT-4 (t=1) | 0.78 | 0.68 | 0.79 | 0.82 | 0.74 | 0.84 | | GPT-3.5 (t=1) | 0.33 | 0.27 | 0.46 | 0.41 | 0.33 | 0.56 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“those generated from the Concise prompt have a high mean LCS value (0.07) and low LED value (11.77).”
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“| Concise (t=0) | 0.70 | 0.53 | 0.71 | 11.77 | 20.55 | 12.14 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 RQ4: Coding Tasks Features and Non-determinism Degree · p. 15
“| R1 (t=0) | 0.18 | 0.00 | 0.00 | 0.00 | 1.20% | |”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the 'Ratio of worst cases', which is the ratio of problems with maximum diff of test pass rate being 1 or OER being 0.”
3 more findings
Create a free account to see every finding of this check.
Citations
Cites
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 5×
“It is one of the most widely used measurement metrics for assessing code generation capabilities 9 [4, 12, 26, 36, 71].”
InfluentialSummary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Self-citationThe Corsini encyclopedia of psychology
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6008 · Cited by 1,082
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 4×
“In our experiment, following the assessment practice of AlphaCode, we use the test set of CodeContests to benchmark the code generation tasks of ChatGPT.”
InfluentialCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Self-citationCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · cited 2×
“Section 8 concludes.”
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07753 · Cited by 767 · cited 2×
“Section 8 concludes.”
Research works printed but not linked: 7
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
- Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
- Andrej Kiviriga. 2023. Efficient Model Checking: The Power of Randomness. Aalborg Universitetsforlag.
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems.
- Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. In The Corsini Encyclopedia of Psychology. John Wiley & Sons, Inc., 1–1.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Technical Report, OpenAI.
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“LLMs are inherently nondeterministic: the same prompt produces different answers on different inference executions (unless the temperature is set to zero, which has often been found to be suboptimal over multiple executions) [9].”
BackgroundInfluentialSelf-citationLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Self-citationCurrent applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“Error analysis [31] Mistakes made during task performance are analyzed to understand metacognitive flexibility.”
BackgroundKernberg, A., Gold, J. A., & Mohan, V. (2024) · Journal of Medical Internet Research · doi:10.2196/54419 · Cited by 120
Chow, J. C. L., & Li, K. (2024) · JMIR Bioinformatics and Biotechnology · doi:10.2196/64406 · Cited by 75
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Azimi, I., Qi, M., Li, W., Rahmani, A. M., & Li, Y. (2025) · Scientific Reports · doi:10.1038/s41598-024-85003-w · Cited by 76
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Shin, J., Tang, C. J., Mohati, T., Nayebi, M., Wang, S., & Hemmati, H. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00082 · Cited by 181
Rapp, A., Lodovico, C. D., & Di, L. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103471 · Cited by 76
Open Science at the generative AI turn: An exploratory analysis of challenges and opportunities
Hosseini, M., Horbach, S. P. J. M., Holmes, K., & Ross-Hellauer, T. (2025) · Quantitative Science Studies · doi:10.1162/qss_a_00337 · Cited by 40
“As Ouyang et al. (2023) note, such factors can negatively impact “ the reliability and reproducibility of empirical software engineering ” and are a potential “ menace ” to the validity of scientific conclusions.”
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 111
“Good quality requirements need to be accurate and complete , without commission (inclusion of irrelevant or incorrect details) and omission (exclusion of necessary details) defects [1].”
Bias Testing and Mitigation in LLM-based Code Generation
Huang, D., Zhang, J. M., Bu, Q., Xie, X., Chen, J., & Cui, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3724117 · Cited by 173
Self-citationA Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 98
Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments
Atıl, B., Aykent, S., Chittams, A., Fu, L., Passonneau, R. J., Radcliffe, E., Rajagopal, G. R., Sloan, A., Tudrej, T., Ture, F., et al. (2025) · Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2025.eval4nlp-1.12 · Cited by 36
Hassan, A. E., Lin, D., Rajbahadur, G. K., Gallaba, K., Cogo, F. R., Chen, B., Zhang, H., Thangarajah, K., Oliva, G., Lin, J. (., et al. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663849 · Cited by 35
Scaling Up Video Summarization Pretraining with Large Language Models
Argaw, D. M., Yoon, S., Heilbron, F. C., Deilamsalehy, H., Bui, T., Wang, Z., Dernoncourt, F., & Chung, J. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00796 · Cited by 63
“Additionally, LLMs have been shown to be capable of generating summaries in a variety of different formats, such as bullet points, paragraphs, and even code [29, 48].”
BackgroundLoya, M., Sinha, D., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.241 · Cited by 33
“In a recent study, Ouyang et al. (2023) emphasized the influence of temperature adjustments on LLM’s performance in code generation tasks.”
BackgroundPrandner, D., Wetzelhütter, D., & Hese, S. (2025) · Frontiers in Education · doi:10.3389/feduc.2024.1417900 · Cited by 55
Leveraging GPT-like LLMs to Automate Issue Labeling
Colavito, G., Lanubile, F., Novielli, N., & Quaranta, L. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644903 · Cited by 58
“5-turbo [53] —i.e., the model that powers ChatGPT at the time of this writing.”
BackgroundHallucination Detection in Large Language Models with Metamorphic Relations
Yang, B., Mamun, M. A. A., Zhang, J. M., & Uddin, G. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715735 · Cited by 67
Self-citationMETAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
Hyun, S., Guo, M., & Babar, M. A. (2024) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst60714.2024.00019 · Cited by 67
Performance of 7 Artificial Intelligence Chatbots on Board-style Endodontic Questions
Jalali, P., Mohammad‐Rahimi, H., Wang, F., Sohrabniya, F., Ourang, S. A., Tian, Y., Martinho, F. C., & Nosrat, A. (2025) · Journal of Endodontics · doi:10.1016/j.joen.2025.06.014 · Cited by 35
Using model-driven engineering to automate software language translation
Lano, K., & Siala, H. A. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00419-y · Cited by 34
ChatGPT in foreign language lesson plan creation: Trends, variability, and historical biases
Dornburg, A., & Davin, K. J. (2024) · ReCALL · doi:10.1017/s0958344024000272 · Cited by 55
Methodology for Code Synthesis Evaluation of LLMs Presented by a Case Study of ChatGPT and Copilot
Ságodi, Z., Siket, I., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3403858 · Cited by 31
Technology adoption performance evaluation applied to testing industrial REST APIs
Poth, A., Rrjolli, O., & Arcuri, A. (2024) · International Conference on Automated Software Engineering · doi:10.1007/s10515-024-00477-2 · Cited by 25
Nanua, S., Steward, R., Neely, B., Datto, M., & Youens, K. E. (2025) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2025.100520 · Cited by 25
Guiding Enumerative Program Synthesis with Large Language Models
Li, Y., Parsert, J., & Polgreen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65630-9_15 · Cited by 66
Turning Low-Code Development Platforms into True No-Code with LLMs
Hagel, N., Hili, N., & Schwab, D. (2024) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3652620.3688334 · Cited by 30
Simplifying software compliance: AI technologies in drafting technical documentation for the AI Act
Sovrano, F., Hine, E., Anzolut, S., & Bacchelli, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10645-x · Cited by 35
“Despiteattemptstomitigateunpredictabilitybysettingcertainparameters (liketemperatureandtop-p)tozero,themodel’snon-deterministicnatureremainsaproblem (Ouyang et al. 2025).”
Enhancing Large Language Models-Based Code Generation by Leveraging Genetic Improvement
Pinna, G., Ravalico, D., Rovito, L., Manzoni, L., & Lorenzo, A. D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56957-9_7 · Cited by 26
Pinto, G., De Souza, C., Rocha, T., Steinmacher, I., Souza, A., & Monteiro, E. (2024) · Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI · doi:10.1145/3644815.3644949 · Cited by 17
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 35
Blanchard, E. G., & Mohammed, P. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64302-6_31 · Cited by 26
The bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
“Conversely, for t > 1 , high probabilities are decreased and low probabilities increased, enhancing randomness. should theoretically be produced deterministically (Ouyang et al., 2025; Ma et al., 2024b).”
Khanghah, K. N., Wang, Z., & Xu, H. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4066095 · Cited by 19
Intelligent application of large language model to life cycle assessment methodology
Zhang, X., Guo, X., Zhao, J., Xiong, J., & Tian, Y. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.146776 · Cited by 15
Impacts of the Usage of Generative Artificial Intelligence on Software Development Process
Santos, P. D. O., Figueiredo, A. C., Moura, P., Diirr, B., Alvim, A. C. F., & dos Santos, R. P. (2024) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3658271.3658337 · Cited by 22
GPT-4 Performance, Nondeterminism, and Drift in Genetic Literature Review
Aronson, S., Machini, K., Shin, J., Sriraman, P., Hamill, S., Henricks, E. R., Mailly, C. J., Nottage, A. J., Amr, S. S., Oates, M., et al. (2024) · NEJM AI · doi:10.1056/aics2400245 · Cited by 16
Ito, S., Muraoka, K., & Nakayama, A. (2025) · Chemistry of Materials · doi:10.1021/acs.chemmater.4c02726 · Cited by 17
Exploring ChatGPT for Toxicity Detection in GitHub
Mishra, S., & Chatterjee, P. (2024) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1145/3639476.3639777 · Cited by 41
“LLM generated outputs can be non-deterministic [40].”
Towards Taming Large Language Models with Prompt Templates for Legal GRL Modeling
de Kinderen, S., & Winter, K. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_17 · Cited by 21
Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories
Tafreshipour, M., Imani, A., Huang, E., de Almeida, E. S., Zimmermann, T., & Ahmed, I. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00106 · Cited by 52
“Unlike traditional software, prompts require testing methods that accommodate their natural language form and the non-deterministic nature of LLMs [39].”
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 70
How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?
Zhang, Y., Song, W., Ji, Z., Yao, D. D., & Meng, N. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3632765 · Cited by 75
“LLMs exhibit inherent non-deterministic behavior [57], [81], [85], causing identical prompts to produce varying outputs.”
Yang, J., & Wang, Y. N. (2024) · IEEE Access · doi:10.1109/access.2024.3366803 · Cited by 16
“This variability poses a potential threat to the validity of scientific conclusions unless researchers adapt their methods to account for it in their empirical analyses [23].”
BackgroundAgile model-driven re-engineering
Lano, K., Haughton, H., Yuan, Z., & Alfraihi, H. (2024) · Innovations in Systems and Software Engineering · doi:10.1007/s11334-024-00568-z · Cited by 18
Neurosymbolic Repair of Test Flakiness
Chen, Y., & Jabbarvand, R. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680369 · Cited by 17
Large Language Models for Automated Web-Form-Test Generation: An Empirical Study
Li, T., Cui, C., Huang, R., Towey, D., & Ma, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735553 · Cited by 37
Statically Contextualizing Large Language Models with Typed Holes
Blinn, A., Li, X., Kim, J., & Omar, C. (2024) · Proc. ACM Program. Lang. · doi:10.1145/3689728 · Cited by 25
Ke, P. F., & Ng, K. C. (2024) · ACM Transactions on Management Information Systems · doi:10.1145/3700597 · Cited by 18
An Exploratory Study on How Non-Determinism in Large Language Models Affects Log Parsing
Astekin, M., Hort, M., & Moonen, L. (2024) · doi:10.1145/3643661.3643952 · Cited by 18
“Going a step beyond the smallest perturbations, Ouyang et al. [17] investigated the behavior of ChatGPT when faced with an identical prompt repeatedly.”
BackgroundResultInfluentialSearch-Based Optimisation of LLM Learning Shots for Story Point Estimation
Tawosi, V., Alamir, S., & Liu, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48796-5_9 · Cited by 9
“Note that according to Ouyang et al. [6] this does not eliminate the risk of non-deterministic response, but minimises it.”
BackgroundLessons from Building StackSpot AI: A Contextualized AI Coding Assistant
Pinto, G., De Souza, C., Neto, J. B., Souza, A., Gotto, T., & Monteiro, E. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3639477.3639751 · Cited by 12
“(1) Inconsistency in the responses : Large language models exhibit inconsistency in their responses [19], sometimes delivering accurate answers while at other times producing random or seemingly clueless information due to their lack of genuine understanding.”
BackgroundTranslating Between SQL Dialects for Cloud Migration
Zmigrod, R., Alamir, S., & Liu, X. (2024) · 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3639727 · Cited by 10
A Roadmap for Software Testing in Open-Collaborative and AI-Powered Era
Wang, Q., Wang, J., Li, M., Wang, Y., & Liu, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709355 · Cited by 26
Schwitter, N. (2025) · Methodological Innovations · doi:10.1177/20597991251313876 · Cited by 18
Large language model based mutations in genetic improvement
Brownlee, A. E. I., Callan, J., Even-Mendoza, K., Geiger, A., Hanna, C., Petke, J., Sarro, F., & Sobania, D. (2025) · Automated Software Engineering · doi:10.1007/s10515-024-00473-6 · Cited by 19
Prompting the Machine: Introducing an LLM Data Extraction Method for Social Scientists
Foisy, L.-O. M., Proulx, É., Cadieux, H., Gilbert, J., Rivest, J., Bouillon, A., & Dufresne, Y. (2025) · Social Science Computer Review · doi:10.1177/08944393251344865 · Cited by 13
Outcomes, Perceptions, and Interaction Strategies of Novice Programmers Studying with ChatGPT
Penney, J., Acharya, P., Hilbert, P., Parekh, P., Sarma, A., Steinmacher, I., & Gerosa, M. A. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736625 · Cited by 17
Prompt Alchemy: Automatic Prompt Refinement for Enhancing Code Generation
Ye, S., Sun, Z., Wang, G., Guo, L., Liang, Q., Li, Z., & Liu, Y. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3589634 · Cited by 27
“In the experiments, the temperature parameter is set to 0 for both code generation and code translation, a common practice when employing large language models for such tasks [11], [12], [41].”
Kataoka, Y., So, R., Banno, M., Kumasawa, J., Someko, H., Taito, S., Terasawa, T., Tsujimoto, Y., Tsutsumi, Y., Wada, Y., et al. (2023) · medRxiv · doi:10.1101/2023.10.31.23297818 · Cited by 8
AgentBound: Securing Execution Boundaries of AI Agents
Bühler, C., Biagiola, M., Di Grazia, L., & Salvaneschi, G. (2026) · Proceedings of the ACM on Software Engineering · doi:10.1145/3808103 · Cited by 9
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
“5 and GPT-4 are non-deterministic even for temperature zero (Chann, 2023; Ouyang et al., 2023).”
BackgroundForgetful Large Language Models: Lessons Learned from Using LLMs in Robot Programming
Chen, J.-T., & Huang, C. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27721 · Cited by 14
Dangi, D. B., Dangi, B. B., & Steinbock, O. (2025) · ACS Omega · doi:10.1021/acsomega.5c01150 · Cited by 7
How Effective are LLMs for Data Science Coding? A Controlled Experiment
do Nascimento, N. M., Guimarães, E. T., Chintakunta, S. S., & Boominathan, S. A. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00041 · Cited by 34
“While prior studies [18] suggest evaluating the best of five attempts to address LLM non-determinism, they primarily assessed code similarity rather than functionality.”
A comparison of large language models and model-driven reverse engineering for reverse engineering
Siala, H. A., & Lano, K. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1516410 · Cited by 11
SE Perspective on LLMs: Biases in Code Generation, Code Interpretability, and Code Security Risks
Krasniqi, R., Xu, D., & Vieira, M. P. A. (2025) · ACM Computing Surveys · doi:10.1145/3774324 · Cited by 14
Large Language Models for Software Engineering: A Systematic Mapping Study
Görmez, M. K., Yılmaz, M., & Clarke, P. M. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71139-8_5 · Cited by 18
Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models
Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3719351 · Cited by 17
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 31
Cognitive phantoms in large language models through the lens of latent variables
Peereboom, S., Schwabe, I., & Kleinberg, B. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100161 · Cited by 8
The Convergence of AI/ML and DevSecOps: Revolutionizing Software Development
Pakalapati, N., Venkatasubbu, S., & Sistla, S. M. K. (2023) · Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online) · doi:10.60087/jklst.vol2.n2.p212 · Cited by 6
CSLLM: Code-Specific Large Language Models—A Survey
Umre, J., Parihar, A. S., & Gupta, A. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130991 · Cited by 10
Leveraging Ethical Narratives to Enhance LLM ‐ AutoML Generated Machine Learning Models
Nelson, J., Pavlidis, M., Fish, A., Polatidis, N., & Manolopoulos, Y. (2025) · Expert Systems · doi:10.1111/exsy.70072 · Cited by 5
“Meanwhile, other research has gone beyond analysing the code itself, instead focusing on the evaluation conditions and metrics used to assess the performance of the LLM-generated code (Liu et al. 2023; Ouyang et al. 2025).”
Using language models to label clusters of scientific documents
Murray, D., Ni, C., Gu, W., & Hubbard, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05445-5 · Cited by 8
“We note that whereas characteristic labeling follows a systematic and deterministic procedure, language models tend to introduce stochasticity into their responses (Ouyang, Zhang, Harman, & Wang, 2024) so responses are non-deterministic such that a quality label cannot be ensured by inputs.”
Merchant, S. A., Merchant, N., Varghese, S. L., & Shaikh, M. J. S. (2025) · World Journal of Radiology · doi:10.4329/wjr.v17.i11.114754 · Cited by 6
Enhancing Autonomous Task Execution in Social Robots with Large Language Models
Becerra, L. R., Colmenares, J. A. R., & Manrique, R. (2024) · International Conference on Automation, Robotics and Applications · doi:10.1109/icara60736.2024.10552978 · Cited by 6
“This lack of determinism has been manifested across various code generation benchmarks [8].”
BackgroundInvestigating LLM Variability in Personalized Conversational Information Retrieval
Lupart, S., Dijk, D. V., Langezaal, E., van Dort, I., & Aliannejadi, M. (2025) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3767695.3769502 · Cited by 8
Towards AI for Software Systems
Eskandani, N., & Salvaneschi, G. (2024) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3664646.3664767 · Cited by 7
Oracle-Guided Program Selection from Large Language Models
Fan, Z., Ruan, H., Mechtaev, S., & Roychoudhury, A. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680308 · Cited by 24
Li, Y., Fu, X., Verma, G., Buitelaar, P., & Liu, M. (2025) · arXiv · doi:10.48550/arxiv.2510.24476 · Cited by 4
“Initially known for their exceptional performance in Natural Language Processing (NLP) tasks, LLMs have since demonstrated a remarkable capacity to generalize beyond language, influencing domains such as code generation [103], reasoning [150], and even scientific discovery [58].”
Comparative Analysis of AI Models for Python Code Generation: A HumanEval Benchmark Study
Bayram, A., Dalveren, G. G. M., & Derawi, M. O. (2025) · Applied Sciences · doi:10.3390/app15189907 · Cited by 9
EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
Shen, Z., He, Y., Wang, Z., Zhang, Y., Sun, G., Ye, W., & Li, A. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729141 · Cited by 21
Hallucination in LLM-Based Code Generation: An Automotive Case Study
Pavel, M., Petrović, N., Mazur, L., Zolfaghari, V., Pan, F., & Knoll, A. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391125 · Cited by 7
“5 [1], GPT-4 [2], CodeX [3], CodeBERT [4], CodeGPT [5] and CodeLLaMA [6] have become increasingly valuable due to their ability to generate code snippets, refactor programs and provide programming assistance.”
Wang, M., Chen, Y., Huang, X., & Lai, Y. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100568 · Cited by 12
Self-citationZhang, Z., & Valeo, C. (2026) · AI for Engineering · doi:10.3390/aieng1010005 · Cited by 6
“…all LLM-driven workflows, including aiswmm and other agentic run-times such as Codex, Claude Code, and OpenClaw, inherently translate natural language prompts into tool calls in non-deterministic ways [55,56], which may lead to subtle but real inconsistencies in the executed workflow [57].”
Raihauti, T., Midingoyi, C. A., Adam, M., Berg‐Mohnicke, M., Brown, H., Cichota, R., Enders, A., Ferreira, T. B., Fumagalli, D., Holzworth, D. P., et al. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111996 · Cited by 4
Detecting Redundancies Between User Stories with Graphs and Large Language Models
Hofmann, L. S., Lauer, A., Kosiol, J., Kesper, A., Wieber, P., Rabieyan, A., & Taentzer, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88531-0_18 · Cited by 4
On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT
Colombo, G., Mariani, L., Micucci, D., & Riganelli, O. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00060 · Cited by 8
“However, the recommendation of using high-temperature values to avoid unwanted plagiarism has to be carefully balanced with settings recommended for code generation, where low-temperature values tend to produce better results [16], [25], [26].”
Do Current Language Models Support Code Intelligence for R Programming Language?
Zhao, Z., & Fard, F. H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3735635 · Cited by 10
Cheng, J., Liu, F., Wu, C., & Zhang, L. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755925 · Cited by 9
Replicating a High-Impact Scientific Publication Using Systems of Large Language ModelsPreprint
Bersenev, D., Yachie‐Kinoshita, A., & Palaniappan, S. K. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.08.588614 · Cited by 5
“Most notably, the nondeter-minism of code-generation tasks [17–19], which stymied the entire pipeline.”
BackgroundLLM-Assisted Requirements Engineering in Agile MDD: Industry Insights and Validation
Spijkman, T., Molenkamp, B., Beudeker, S., Overbeek, S. J., & Dalpiaz, F. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00043 · Cited by 3
“Focus areas include code generation [12], [13], testing [14], and code summarization [15].”
Metadata
- Authors
- Ouyang, S., Zhang, J. M., Harman, M., & Wang, M.
- Affiliations
- University of London (GB)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-01-22
- Volume
- 34
- Issue
- 2
- Pages
- 1-28
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402860127
- Semantic Scholar
- dc7de606ecf7c65abc26396629b633e18d8704d5
- Cited by
- 258
- References
- 41
- Citations by year
- 2026: 108 · 2025: 90
- FWCI
- 94.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Explore Bristol Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- arXiv (Cornell University) (public-domain)
- Funders
- UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (EP/S023356/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- ChatGPT, large language models, code generation, non-determinism, reproducibility, temperature parameter, semantic similarity, syntactic similarity, structural similarity, HumanEval, APP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.