An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 6 of the 45 sources its claims cite were checked for support.
Abstract
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.
Overview
CoChat Summary
TL;DR
This paper empirically studies non-determinism in ChatGPT code generation across 829 problems from CodeContests, APPS, and HumanEval. It finds substantial semantic, syntactic, and structural variation across repeated identical prompts, and shows that temperature 0 does not guarantee determinism.
Key findings
- Under default settings, the ratio of coding tasks with zero equal test output among five requests was 75.76% for CodeContests, 51.00% for APPS, and 47.56% for HumanEval; maximum test-pass-rate difference reached 1.00 on all three datasets and accounted for 39.63% of HumanEval problems.
- Setting temperature to 0 reduced but did not eliminate non-determinism: there were still 43.64% (CodeContests), 27.40% (APPS), and 18.29% (HumanEval) of problems with no equal test output among five candidates.
- GPT-4 was slightly more deterministic than GPT-3.5 at temperature 1, but their non-determinism was similar at temperature 0; Chain-of-Thought prompting led to more non-determinism at temperature 0, while prompts requesting concise code were more stable.
- In a manual review of 76 LLM-based code generation papers from the last two years, only 21.1% (16/76) considered non-determinism in their experimental evaluation.
Methods
- Empirical study of 829 code generation problems: CodeContests (165), APPS (500 randomly sampled), and HumanEval (164), using ChatGPT API models `gpt-3.5-turbo-0125` and `gpt-4-0613` with zero-shot Python3 prompts.
- For each task, the API was called five times with the same prompt; generated code was extracted, compiled, and run against benchmark test suites with a 3-second timeout.
- Non-determinism was measured via semantic similarity (test pass rate, output equivalence rate), syntactic similarity (Longest Common Subsequence, Levenshtein Edit Distance), and structural similarity (AST-based `pycode_similar` with Unified_Diff and Tree_Diff), with Kruskal-Wallis tests for significance.
Limitations
- The paper states that it does not study different programming languages, uses only a few code similarity measures, does not fully consider prompt influence on non-determinism, and focuses exclusively on ChatGPT rather than other LLMs.
- It also states that measuring semantic similarity through test execution outputs may oversimplify semantic similarity by ignoring code meaning, logic, and diverse correct solutions.
- The text makes evident that several analyses (RQ2, RQ3, RQ5, RQ6) report main results only for CodeContests, with other dataset results placed on the homepage, limiting the in-paper generalizability of those comparisons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 2 Not valid · 18 Not checked · 125 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
What the sources say
- Partly supportedcites [15] [15] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. arXiv:2310.03533 [cs.SE] https: //arxiv.org/abs/2310.03533 - sourceExperimental — accuracy not yet measured
Our copy treats ChatGPT as a dominant subject of code-generation research — the background section ("A Large Language Model (LLM) refers to...", p. n/a) says the step change in GPT-3/3.5 performance attracted "a great deal of interest in GPT (and ChatGPT) in particular", and Section D (Scientific Evaluation of LLM-based Code Generation) surveys ChatGPT-centred work (Borji's failure analysis, Ouyang et al.'s non-determinism study, Yetistiren et al.'s comparison). However, the paper never states the superlative the claim attributes to it: it reports no counts or ranking of how often each LLM is used across the code-generation literature, and the only adoption figure it gives (Section C, "46% of the developers' code was written by Copilot", i.e. Codex/GPT-4-based Copilot in industry) is about practitioner use, not literature uptake. So the "widely used/much-studied" part is supported while "the most widely adopted" is not established by this source. (LLM, conf=0.68)
- Not checked yetcites [16] [16] Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE Computer Society, Los Alamitos, CA, USA, 876-885. https://doi.org/10.1109/COMPSAC57700.2023.00117 - source
- Partly supportedcites [23] [23] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2023. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. arXiv:2309.08221 [cs.SE] https: //arxiv.org/abs/2309.08221 - sourceExperimental — accuracy not yet measured
Section 2.2 identifies ChatGPT as an LLM and describes this paper's use of it for code refinement. Related Work says ChatGPT has been evaluated on several software-engineering tasks. Neither passage establishes that it is the most widely adopted LLM in the code-generation literature. (LLM, conf=0.83)
- Partly supportedcites [40] [40] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 - sourceExperimental — accuracy not yet measured
The Additional information section includes ChatGPT among 26 models described as representative and popular for evaluation, and the paper evaluates its code generation. It does not compare adoption across the literature or establish that ChatGPT is the most widely adopted LLM for code generation. (LLM, conf=0.84)
- Partly supportedcites [42] [42] Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1-26. - sourceExperimental — accuracy not yet measured
Section 8.1 says ChatGPT has gained popularity for code generation, and the Introduction describes its rapid growth in users. Neither passage establishes that it is the most widely adopted LLM for code generation in the literature; the cited paper provides no comparison of adoption across models or studies. (LLM, conf=0.85)
- Partly supportedcites [63] [63] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Trans. Softw. Eng. 50, 4 (Feb. 2024), 911-936. https: //doi.org/10.1109/TSE.2024.3368208 - sourceExperimental — accuracy not yet measured
Section 5.1 LLM Models states that 'the most commonly utilized LLM in software testing tasks is ChatGPT' and that it 'ranks in the top position in terms of our collected studies' (102 papers), so the source does support ChatGPT being the most adopted LLM in the software-testing literature it surveyed. However, the claim's qualifier 'in code generation' is not what the source reports: the Introduction explicitly separates the two domains ('software testing possesses unique characteristics that differentiate it from code generation'), and Section 7 credits the NL2Code/code-generation survey to Zan et al. [168], a different work. The ranking is also bounded to the authors' collected studies rather than the whole literature. (LLM, conf=0.78)
- Not checked yetcites [70] [70] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24) . Association for Computing Machinery, New York, NY, USA, Article 37, 12 pages. https://doi.org/10.1145/3597503.3623316 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, non-determinism affects the reliability and reproducibility of empirical software engineering [53].”
What the sources say
- Could not verifycites [53] [53] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv:2201.11227 [cs.LG] https: //arxiv.org/abs/2201.11227 - sourceExperimental — accuracy not yet measured
Our copy of Synchromesh (Poesia et al., 2022) is about the reliability of LLM code generation — i.e., preventing syntactic/semantic implementation errors via Target Similarity Tuning and Constrained Semantic Decoding (Abstract p.1; Intro p.1-2; Discussion p.8-9) — not about non-determinism undermining empirical software engineering. The nearest content is its use of stochastic sampling (temperature τ = 0.7, Valid@K, Results p.8) and the observation that LLMs are sensitive to which few-shot examples are in the prompt (Intro p.2), but no passage frames non-determinism as a threat to reliability or reproducibility of empirical software engineering. Searched: 'non-determinism / reproducibility / empirical software engineering', 'stochastic sampling temperature reproducibility of experiments', 'deterministic decoding / outputs vary across runs', 'validity of experiments'; the copy is not truncated, so the point is simply absent rather than missing. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for three different code generation datasets (i.e., CodeContests, APPS, and HumanEval), respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we collect 76 LLM-based code generation papers that appeared in the last 2 years. Our manual analysis results highlight that only 21.1% of these papers consider the non-determinism threat in their experiments.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“accounts for 39.63% of the problems in HumanEval, the most widely used code generation benchmark.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
80 more claims
Create a free account to see all 146 claims and to verify the 38 references still open.
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
- Probably not a substantial claim. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One major issue is the inherent stochasticity in the models’ architecture, particularly in transformer-based models, which utilize probabilistic techniques during inference (e.g., beam search or temperature sampling)125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, Ouyang et al. reported that only a minority of studies take non-determinism into account in their experimental evaluation when using ChatGPT for code generation, suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent work has identified multiple sources of non-determinism in large language models.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
The cited source says what the claim attributes to it.
In the citing paper
“For code generation specifically, Ouyang et al. (2025) find that repeated queries yield different implementations even with greedy sampling.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.2 RQ2: Influence of Temperature · p. 13
“When temperature =0.5, we observe that ChatGPT tends to generate code candidates that are more deterministic than temperature=1, but less deterministic than temperature=0.”
§ 4.2 RQ2: Influence of Temperature · p. 13
“| 0 | 0.41 | 0.39 | 0.67 | 0.50 | 0.46 | 0.74 | | 0.5 | 0.61 | 0.49 | 0.63 | 0.69 | 0.58 | 0.71 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 1 Introduction · p. 2
“the ratio of problems with not a single equal test output among the top-five code candidates is above 50% for all the benchmarks we study”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the ratios of tasks with zero equal test output (i.e., OER=0) among the five code candidates are 75.76%, 51.00%, and 47.56% for the three datasets, respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“For temperature=1, we can observe that GPT-4 is slightly more deterministic than GPT-3.5, with lower test pass rate variance, lower ratio of worst cases, lower OER and OER (no ex.), lower LCS, higher LED, and lower structural similarity under two settings.”
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“| GPT-4 (t=1) | 0.78 | 0.68 | 0.79 | 0.82 | 0.74 | 0.84 | | GPT-3.5 (t=1) | 0.33 | 0.27 | 0.46 | 0.41 | 0.33 | 0.56 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“those generated from the Concise prompt have a high mean LCS value (0.07) and low LED value (11.77).”
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“| Concise (t=0) | 0.70 | 0.53 | 0.71 | 11.77 | 20.55 | 12.14 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 RQ4: Coding Tasks Features and Non-determinism Degree · p. 15
“| R1 (t=0) | 0.18 | 0.00 | 0.00 | 0.00 | 1.20% | |”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the 'Ratio of worst cases', which is the ratio of problems with maximum diff of test pass rate being 1 or OER being 0.”
3 more findings
Create a free account to see every finding of this check.
Citations
Cites
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 5×
“It is one of the most widely used measurement metrics for assessing code generation capabilities 9 [4, 12, 26, 36, 71].”
InfluentialSummary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Self-citationThe Corsini encyclopedia of psychology
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6008 · Cited by 1,082
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 4×
“In our experiment, following the assessment practice of AlphaCode, we use the test set of CodeContests to benchmark the code generation tasks of ChatGPT.”
InfluentialCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Self-citationCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · cited 2×
“Section 8 concludes.”
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07753 · Cited by 767 · cited 2×
“Section 8 concludes.”
Research works printed but not linked: 7
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
- Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
- Andrej Kiviriga. 2023. Efficient Model Checking: The Power of Randomness. Aalborg Universitetsforlag.
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems.
- Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. In The Corsini Encyclopedia of Psychology. John Wiley & Sons, Inc., 1–1.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Technical Report, OpenAI.
Cited by
Items 101–200 of 262
Kim, H., Jung, K., Shin, S.-H., Lee, W., Lee, J. H., Park, H., & Choi, Q. (2025) · Scientific Reports · doi:10.1038/s41598-025-20066-x · Cited by 7
“This residual variability can be attributed to floating-point precision limits and nonassociative arithmetic operations in hardware implementations— factors that align with the minimal variability observed in our results 26,27 .”
Next output is better? Overgeneration effect in generative artificial intelligence interaction
Yang, H. Y., Zheng, H., Lu, L., & Hu, P. (2025) · Information & Management · doi:10.1016/j.im.2025.104294 · Cited by 5
Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges
Nashid, N., Ding, D., Gallaba, K., Hassan, A. E., & Mesbah, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00137 · Cited by 15
“While decoding is configured to be deterministic at the API level, generation remains non-deterministic in practice due to low-level system factors such as thread scheduling, memory layout, and backend randomness [63].”
Reentrancy Detection in the Age of LLMs
Ressi, D., Spanò, A., Rizzo, M., Benetollo, L., & Rossi, S. (2026) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn69566.2026.00050 · Cited by 5
“LLMs were evaluated zero-shot with a temperature of 0 to ensure deterministic outputs and reproducibility across runs, following empirical evaluations of LLM-based program analysis [45].”
Vito, G. D., D’Amici, D., Izzo, F., Ferrucci, F., & Nucci, D. D. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00024 · Cited by 8
“Ouyang et al. [29] found significant differences in correctness, test outputs, syntax, and code structure from the same instruction across multiple requests in ChatGPT’s code generation.”
State-Based Dynamic Graph with Breadth First Progression For Autonomous Robots
Chugh, T., Tyagi, K., Srinivasan, P., & Challagundla, J. (2024) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc60891.2024.10427646 · Cited by 5
Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows
Shih, J. Y., Mohanty, V., Katsis, Y., & Subramonyam, H. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651115 · Cited by 4
Leveraging Generative Artificial Intelligence to Broaden Participation in Computer Science
Jayachandran, D., Maldikar, P. S., Love, T. S., & Blum, J. J. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31262 · Cited by 3
“In addition, the role of randomness in output will be described in the context of these systems, for example, the role of temperature in ChatGPT (Ouyang et al. 2023).”
BackgroundA Comparative Analysis of Model Alignment Regarding AI Ethics Principles
Palumbo, G., Carneiro, D. R., & Alves, V. (2024) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-66635-3_27 · Cited by 6
Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multiagent Control
Anne, T., Syrkis, N., Elhosni, M., Turati, F., Legendre, F., Jaquier, A., & Risi, S. (2025) · IEEE Transactions on Games · doi:10.1109/tg.2025.3564042 · Cited by 12
“We concluded that high temperatures increase the chance of returning an invalid plan and settled on a minimal temperature of 0 (except for oi-mini , which was a preview with a fixed temperature of 1).”
Shin, E. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70089 · Cited by 10
Exploration of Computer Programming Teaching Reform Based on Large Language Models
Zhao, Y., Ling, X., Zhu, H., Zhou, F., & Deng, Z. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00062 · Cited by 2
Collaborating with LLMs Through a Voice and Graphical User Interface
Sorokin, L., Huynh, K., Eiband, M., Stappen, L., & Dillmann, J. (2025) · International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI Adjunct) · doi:10.1145/3737821.3749555 · Cited by 4
SecLLM: Enhancing Security Smell Detection in IaC With Large Language Models
Vito, G. D., Palomba, F., & Ferrucci, F. (2025) · IEEE Access · doi:10.1109/access.2025.3637505 · Cited by 5
CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews
Sorathiya, A., & Ginde, G. (2025) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew66121.2025.00015 · Cited by 6
“5-turbo [27].”
Evaluating and Defending Against Adversarial Attacks on LLM-Generated LSTM Models
Ghahremani, M., Podshyvalin, A., & Taheri, R. (2026) · Springer eBooks · doi:10.1007/978-3-031-99447-0_13 · Cited by 2
On the Promises and Challenges of AI-Powered XR Glasses as Embodied Software
Gu, R., Zhang, J., Rojas, J. M., & Shin, D. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00030 · Cited by 4
“XAI outputs should be reliable and aligned with users’ needs, despite FMs’ non-determinism [38] and limited interpretability.”
Self-citationAdministrative Decision-Making with Generative AI: The Challenge of Epistemic Boundedness
Kim, Y., Kim, J., Kim, T., & Cho, H.-C. (2026) · Administration & Society · doi:10.1177/00953997251409156 · Cited by 5
“LLMs then undergo fine-tuning or post-training to better align their behavior with human preferences and make them more useful for practical applications (Chung et al., 2024; Ouyang et al., 2025; Wei et al., 2022).”
Measuring LLM Code Generation Stability via Structural Entropy
Song, Y., Sun, T., Tang, X., Rajput, P. K., Bissyandé, T. F., & Klein, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00343 · Cited by 7
“For example, ChatGPT produces completely different code on repeated queries for the same programming task in the majority of cases (e.g. 75.8% of tasks showed zero identical test outputs across runs) [2].”
InfluentialFlakyGuard: Automatically Fixing Flaky Tests at Industry Scale
Li, C., Behrang, F., Shi, A., & Liu, P. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00179 · Cited by 2
“LLM non-determinism represents a potential threat to evaluation validity [29].”
Evaluating the Accuracy and Stability of Frontier LLMs on ENADE Computer Science Questions
Carvalho, L. D. M., Carvalho, C. M., & Mendonça, N. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15993-9_18 · Cited by 2
Agent-Supported Foresight for AI Systemic Risks: AI Agents for Breadth, Experts for Judgment
Fröhling, L., Giaconia, A., Bogucka, E. P., & Quercia, D. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790712 · Cited by 2
Peters, U., Bertazzoli, A., DeJesus, J. M., van der Velden, G. J., & Chin-Yee, B. H. (2026) · Public Understanding of Science · doi:10.1177/09636625261425891 · Cited by 4
“UIs incorporate system-level parameters that providers (OpenAI, etc.) do not disclose (e.g. response randomness or moderation features) and that may affect output stability (Ouyang et al., 2025).”
Comparing LLM and expert assessments of journal quality
Saarela, M., Pölönen, J., Linna, A.-K., Wahlfors, L., Correia, A., & Kärkkäinen, T. (2026) · Scientometrics · doi:10.1007/s11192-026-05644-8 · Cited by 3
“To maintain reproducibility and reduce stochastic variation, all models were run with temperature = 0.0 , although prior work indicates that some nondeterminism may persist in LLM outputs (Ouyang et al., 2025).”
InfluentialKafagi, A.-R., Atassi, N., & Kafagi, A.-H. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-108775 · Cited by 2
“As LLMs continue to evolve at pace, ongoing evaluation against validated specialist benchmarks will be essential to ensure their safe, effective and evidence-based integration into surgical education.”
An Empirical Comparison of Code Generation Approaches for Ansible
Darnell, B. L., Chopra, H., Councilman, A., Grove, D., Wang, Y.-X., & Adve, V. (2024) · doi:10.1145/3643661.3643951 · Cited by 5
Can Test Generation and Program Repair Inform Automated Assessment of Programming Projects?
Gu, R., Rojas, J. M., & Shin, D. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988955 · Cited by 2
“Inconsistent responses, like generated code, can undermine reliability during the software development process [32].”
InfluentialStudying How Configurations Impact Code Generation in LLMs: The Case of ChatGPT
Donato, B., Mariani, L., Micucci, D., & Riganelli, O. (2025) · IEEE International Conference on Program Comprehension · doi:10.1109/icpc66645.2025.00055 · Cited by 17
“The temperature has been specifically studied also in Ouyang et al. [17], again confirming that low values may guarantee more consistent results from GPT models.”
InfluentialTotal Recall? Evaluating the Macroeconomic Knowledge of Large Language Models
Crane, D. L., Karra, A., & Soto, P. E. (2025) · Finance and Economics Discussion Series · doi:10.17016/feds.2025.044 · Cited by 4
“Our results generally mirror those of Ouyang et al. (2025), who show significant non-determinism in OpenAI’s GPT-3.”
Evaluation of the Code Generated By Large Language Models: The State of the Art
Ying, Z., Towey, D., & Zhang, Y. (2025) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac65507.2025.00064 · Cited by 3
On Developers’ Self-Declaration of AI-Generated Code: An Analysis of Practices
Kashif, S. M., Liang, P., & Tahir, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771937 · Cited by 17
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7 · Cited by 5
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v2 · Cited by 1
On Randomness in Agentic Evals
Bjarnason, B. H., Silva, A., & Monperrus, M. (2026) · arXiv · doi:10.48550/arxiv.2602.07150 · Cited by 1
Performance of Large Language Models on the Radiation and Cancer Biology Practice Exam
Bertschmann, J., Xu, Y., Bayley, C., Abdellatif, A., & Lee, S. L. (2024) · Frontiers in Oncology · doi:10.1016/j.ijrobp.2024.07.1347 · Cited by 1
SecureMind: A Framework for Benchmarking Large Language Models in Memory Bug Detection and Repair
Wang, H., Jacob, D., Kelly, D., Elkhatib, Y., Singer, J., & Wang, Z. (2025) · ACM SIGPLAN International Symposium on Memory Management (ISMM) · doi:10.1145/3735950.3735954 · Cited by 4
Investigating the Performance of Small Language Models in Detecting Test Smells in Manual Test Cases
Lucas, K., Gheyi, R., Ribeiro, M., Palomba, F., Martins, L., & Soares, E. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.11572 · Cited by 5
Ethical AI prompt recommendations in large language models using collaborative filtering
Nelson, J., Baimagambetov, A., Avgerinakis, K., & Polatidis, N. (2025) · International Journal of Parallel Emergent and Distributed Systems · doi:10.1080/17445760.2025.2573086 · Cited by 1
“Meanwhile, other research [13, 14] has gone beyond analysing the code itself, focusing instead on the evaluation conditions and metrics used to assess the performance of LLM-generated code.”
TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
Tambon, F., Nikanjam, A., Zid, C., Khomh, F., & Antoniol, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3773285 · Cited by 14
“This result echoes similar observations in other studies: Ouyang et al. [63] showed that the test pass rate does not necessarily correlate with difficulty, as aggregated by human participants in a coding contest, and Kou et al. [44] highlighted major differences when comparing how a human developer…”
Feischl, C., & Kern, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3777383 · Cited by 5
GenAI for Robot System Engineering
Clever, D., Tan, R., Stuhlenmiller, F., Maczey, S., & Dai, F. (2025) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case58245.2025.11163987 · Cited by 1
“As analyzed in [17], this does not guarantee deterministic behavior for GPT-4 in practice.”
Prompting Instability: An Empirical Study of LLM Robustness in Code Vulnerability Detection
Han, S., Tan, T., Miao, Y., Chen, X., & Sun, N. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-4969-6_18 · Cited by 6
RISE: Rule-Driven SQL Dialect Translation via Query Reduction
Xie, X., Zhang, Y., Dou, W., Gao, Y., Cui, Z., Song, J., Yang, R., & Wei, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773257 · Cited by 1
The Roboagents Are Coming! The Promise and Challenge of Artificial Intelligence Advance Directives
Appel, J. M. (2026) · The Hastings Center Report · doi:10.1002/hast.70042 · Cited by 1
Clouseau: A Hierarchical Multi-Agent Approach for Autonomous Attack Investigation
Aldaihan, A., Alotaibi, F., & Maffeis, S. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00051 · Cited by 11
Garlough-Shah, G., Park, J. I., Hayati, S. A., Kang, D., & Huh, J. (2026) · Journal of Advertising Research · doi:10.1080/00218499.2025.2589501 · Cited by 3
Lu, Z., Zhang, P., Nie, Y., Yang, Y., Tang, Y., Chong, C. Y., & Zhou, Y. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798251 · Cited by 3
Berndt, A., Bach, T., & Baltes, S. (2026) · doi:10.1145/3786178.3788327 · Cited by 1
Proactive Cyber Defense: A Real-Time CTI Framework with ATT&CK–D3FEND Mapping
Jo, R., Lee, H.-B., Han, J., Jung, W.-K., Lee, J.-Y., Kang, T.-Y., Kim, Y., Kwak, B. I., Han, M. L., & Kang, J. (2026) · Systems · doi:10.3390/systems14050575 · Cited by 2
“Because of the probabilistic nature of the generation process, the narrative style, detailed progression, and included information may vary even from the same seed [38].”
Automated Coding of Communication Data Using LLM: Consistency across Subgroups
Hao, J., Cui, W., Kyllonen, P. C., & Kerzabi, E. (2026) · Journal of Educational Measurement · doi:10.1111/jedm.70049 · Cited by 5
Navigating maritime emergencies with large models: A lifecycle-oriented review
Xu, J., Yang, Q., Yang, Z., Gao, Y., Yang, K., & Jiang, J. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115411 · Cited by 2
MinsC2Rust: LLM-driven project-level code migration from C to safe Rust
Kang, Z., Zhu, Q., 缪文瑞, Wang, B., Zhang, X., & Huang, H. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10905-4 · Cited by 1
“This is because LLMs often exhibit semantic inaccuracies when handling long and overlapping contexts, and as Ouyang et al. (2025) demonstrated in their LLM-based code generation experiments (e.g., with Chat - GPT), these models inherently suffer from non-determinism in code output–even with…”
Agentic Pipelines in Embedded Software Engineering: Emerging Practices and Challenges
Sun, S., & Staron, M. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00024 · Cited by 6
“On the other hand, the non-deterministic behavior of LLMs [20], the lack of transparent provenance, and the evolution of model versions raise concerns about long-term support and the validity of certification [14].”
On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems
Berndt, A., Bach, T., Gemulla, R., Kessel, M., & Baltes, S. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786919 · Cited by 4
Slotkienė, A., & Makaris, L. (2026) · Electronics · doi:10.3390/electronics15163694 · Cited by 1
Ruiz, F. V., Grishina, A., Hort, M., & Moonen, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3771922 · Cited by 20
“Table 7 reports the number of manually verified correct fixes for each model in our approach, as well as in previous work [25].”
Introduction to theme section on requirements formalisation
Lano, K., Kolahdouz-Rahimi, S., Tehrani, S. Y., Burgueño, L., & Umar, M. (2024) · Software & Systems Modeling · doi:10.1007/s10270-024-01241-6 · Cited by 1
Olivos, F., Kamelski, T., & Gac, S. A. (2025) · Teaching Sociology · doi:10.1177/0092055x251397371 · Cited by 4
“…of student identities, the provision of template responses to reduce overreli-ance on training data, and the use of multiple grading runs to establish consistent grading patterns that mitigate the nondeterministic nature of LLMs like the GPT family (Coronado-Blázquez 2025; Ouyang et al. 2025).”
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v3
Ito, S., Muraoka, K., & Nakayama, A. (2024) · ChemRxiv · doi:10.26434/chemrxiv-2024-wqxl7-v4
A Large Language Model Pipeline to Automate the Solution of Competitive Programming Problems
Jayachandran, D., & Blum, J. J. (2024) · ASEE Middle Atlantic Section Spring Conference · doi:10.18260/1-2--45702
The Effects of Semantic Information on LLM-Based Program Repair
Hori, S., Matsumoto, S., Higo, Y., Kusumoto, S., Yasuda, K., Ito, S., & Huyen, P. T. T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78386-9_28
Your Wish Is My Command? Letting GenAI Out of the Bottle to Channel Employee Creativity
Aleksić, D., Hewett, R., & Giessner, S. R. (2025) · Springer eBooks · doi:10.1007/978-3-031-76902-3_17
Architectures of Error: A Philosophical Inquiry into AI and Human Code GenerationPreprint
Sartori, C. C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5265751
Exploring Zero-Shot Prompting for Generating Data Format Descriptions
Anantharaman, P., & Varadharaju, V. (2025) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw67851.2025.00005
“For instance, Ouyang et al. [34] found that ChatGPT exhibits substantial variability in code generation, with a high percentage of tasks producing non-identical outputs even under identical prompts.”
Zeng, Z., Ratnavelu, K., & Huat, O. S. (2025) · WSEAS TRANSACTIONS ON COMPUTER RESEARCH · doi:10.37394/232018.2025.13.42 · Cited by 1
When does ChatGPT refer someone to a plastic surgeon?
Dagi, A. F., Jones, N. E., & Bogue, J. T. (2025) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2025.07.038
From Prompts to Findings: Detecting Command Injection in Python Packages at ScalePreprint
Chen, J., Wang, Y., & Wang, Q. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5442371
GPT-4 as an Automatic Grading System for Open-Ended Questions in Computer Science Education
Heimberg, M., & Bernhard, W. (2025) · IEEE Integrated STEM Education Conference (ISEC) · doi:10.1109/isec64801.2025.11147305
“The OpenAI API parameters were configured to ensure determinism (e.g., temperature set to 0), acknowledging that even with a temperature of 0, absolute determinism is not guaranteed, as reflected in both the results of this work and the literature [13].”
Enhancing Accessibility for Non-experts in Low-Code/No-Code Through the Addition of an AI Chatbot
Troyer, N. D., Croon, R. D., & Verbert, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04999-5_12
Architectures of Error: A Philosophical Inquiry into Human and AI Code
Sartori, C. C. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01056-x · Cited by 2
Sorathiya, A., & Ginde, G. (2025) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew66121.2025.00039 · Cited by 1
“5-turbo [39].”
LLM-driven authentic assessment of machine learning programs in the LearnMatic data science ITS
Jamil, H. M. (2026) · Information Sciences · doi:10.1016/j.ins.2026.123783
A Preliminary Study on Large Language Models Self-Negotiation in Software Engineering
Tao, C., Shu, H., Kondo, M., & Kamei, Y. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00088
“Specifically, researchers and practitioners have investigated their capabilities in code-related tasks (e.g., code generation) [5], [6], [16].”
Prompts Engineering Challenges in Software Code Generation
Gutierrez, Y., Camacho, E., Pardo, C., & Villarreal, V. (2025) · IEEE Congreso Internacional en Inteligencia Ambiental, Ingeniería de Software y Salud Electrónica y Móvil (AmITIC) · doi:10.1109/amitic68284.2025.11214606
“…, focused on generating correct, adaptive and reliable code, ensuring that the proposed solutions are technically accurate and error resistant [4], [19], [21], [28]; (4) Code refinement, repair and enhancement techniques, which include methods to review, correct and optimize model-generated code,…”
Bañuls-Lapuerta, F. A., Marti-Miralles, V., González-García, R. J., & Rico, G. M. (2025) · doi:10.2196/preprints.86786
AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot
Wang, X., Dong, L., Rangasrinivasan, S., Nwogu, I., Setlur, S., & Govindaraju, V. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247695 · Cited by 7
“Performance Variation Analysis: The non-deterministic behavior of LLMs leads to inherent variability in their outputs [36].”
Muteract: Interactive and Iterative Prompt Mutation Interface for LLM Developers and Evaluators
Meena, Y. K., Mondal, S., & Potta, M. P. (2025) · International Conference of Human-Computer Interaction Design & Research · doi:10.1145/3768633.3770129
Cash or Comfort? How LLMs Value Your Inconvenience
Cedro, M., Ichmoukhamedov, T., Goethals, S., He, Y., Hinns, J., & Martens, D. (2026) · Communications of the ACM · doi:10.1145/3799434 · Cited by 3
“Full details appear in Table 2.”
HoarePrompt: Structural Reasoning About Program Correctness in Natural Language
Bouras, D. S., Dai, Y., Wang, T., Xiong, Y., & Mechtaev, S. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773206 · Cited by 16
Song, Y., & Cohen, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-09156-7_2 · Cited by 8
“However, we observed that ChatGPT-4o, with a temperature setting of zero, did not follow greedy decoding, consistent with prior findings [27].”
Exploratory analysis of logical and intuitive reasoning in large language models
Peruchini, M., & Teixeira, J. M. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00942-5 · Cited by 2
“According to Ouyang et al. [23], the discrepancy in performance is attributable to the inherent non-determinism of large language models, which may Prepared by the authors produce different outputs across runs even under fixed conditions due to randomness in model inference.”
Martre, P., Raihauti, T., Midingoyi, C. A., Adam, M., Berg‐Mohnicke, M., Brown, H. E., Cichota, R., Enders, A., Ferreira, T., Fumagalli, D., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6066356
LLMs Choose the Right Stack: From Patterns to Tools
Copei, S., Hohlfeld, O., Kosiol, J., & Ристоски, А. (2025) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00057
“By measuring variance we gauge the indeterminism inherent to each LLM, an important factor because even with temperature = 1.0, stochasticity can prevent reproducible results [22].”
InfluentialGeneralizable Secure Code Generation Framework for Robust Software Development Practices
Sandana Karuppan., A., Shanmugasundaram, S., Mouriya, J., & Sanjeeth, J. (2025) · International Conference on Mobile Networks and Wireless Communications (ICMNWC) · doi:10.1109/icmnwc66779.2025.11354349 · Cited by 1
Detecting and Mitigating Inconsistencies Between Code, Documentation and Tests
Kiecker, T. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00397
“These range from language-specific static analysis tools [14] to general-purpose NLP-driven models trained to classify documentation-code pairs as consistent or inconsistent [2].”
Age matters: Analyzing age-related discussions in app reviews
Nirmani, S., Sharma, G., Khalajzadeh, H., & Shahin, M. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112800 · Cited by 1
“Non-determinism, in simple terms, is the problem of not always producing the same output for the same input [59].”
Automating Access: LLM-Based Permission Request Processing for the Enterprise
Niedermeier, M., Wittges, H., & Rinderle‐Ma, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15140-7_17
Walking on Thin Ice?: Defending RAG-based Code Recommender Systems Against Poisoning AttacksPreprint
Nguyen, P. T., Ngo, Q. M., Vu, D. H., Bui, A. M. T., Ruscio, D. D., & Penta, M. D. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6193142
Lee, D. Y. H., Parker, A. J., Norbury, C., & Shanks, D. R. (2026) · Royal Society Open Science · doi:10.1098/rsos.250381 · Cited by 2
Bañuls-Lapuerta, F. A., Marti-Miralles, V., González-García, R. J., & Rico, G. M. (2026) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/86786 · Cited by 3
“The experimental design was developed with the explicit understanding that current AI systems are inherently nondeterministic and that the replication of results will be inconsistent [51].”
Xu, D., Gondal, I., Yi, X., Sušnjak, T., & McIntosh, T. R. (2026) · ACM Transactions on Internet Technology · doi:10.1145/3799420 · Cited by 1
Lorecast: Layout-Aware Performance and Power Forecasting from Natural Language
Wang, R., Sengupta, P., Roman-Vicharra, C., Chen, Y., & Hu, J. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420573
“Non-determinism is a fundamental challenge underlying the instability of LLM-generated outputs [44] [45].”
Wang, Y., Alejandrez, A., Stewart, C., Malekhedayat, M., Luong, J., Smith‐Bindman, R., & Mongan, J. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01882-7
MedNuggetizer: Confidence-Based Information Nugget Extraction from Medical Documents
Donabauer, G., Ateia, S., Kruschwitz, U., Burger, M., May, M., Gilfrich, C. P., Haas, M., Garzaro, J. R. R., & Eckl, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21321-1_26 · Cited by 1
Software Development Using Transformer-Based LLMs
Вениаминович, Г. П. (2026) · doi:10.1109/elcon-cp69823.2026.11452207
“Experimental studies show that with repeated queries with the same instructions, a model can produce different solutions to the same problem, which reduces reproducibility and complicates the integration of such tools into formalized development processes [8].”
Evaluating GPT ratings of EFL writing: A scoping review
Chen, Y. (2026) · Assessing Writing · doi:10.1016/j.asw.2026.101044 · Cited by 3
Zhu, T., Wen, Z., Liu, S. J., Li, Y., Zhang, T., & Xia, X. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3808230 · Cited by 3
AI Co-CreationPreprint
Hu, M., Jin, Y., & Nejati, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6554898
Neubig, S., Göhl, L., Radhakrishnan, R., Weirich, M., Hein, A., Keller, R., & Krcmar, H. (2026) · Information Technology & Tourism · doi:10.1007/s40558-026-00363-5
“While agents often enhance reasoning power and adaptability compared to purely prompt‑based approaches, their increased autonomy interacts with input sensitivity and the inherent non‑determinism of LLMs (Ouyang et al. 2025), which can produce unstable control flows.”
Metadata
- Authors
- Ouyang, S., Zhang, J. M., Harman, M., & Wang, M.
- Affiliations
- University of London (GB)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-01-22
- Volume
- 34
- Issue
- 2
- Pages
- 1-28
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402860127
- Semantic Scholar
- dc7de606ecf7c65abc26396629b633e18d8704d5
- Cited by
- 258
- References
- 41
- Citations by year
- 2026: 108 · 2025: 90
- FWCI
- 94.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Explore Bristol Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- arXiv (Cornell University) (public-domain)
- Funders
- UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (EP/S023356/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- ChatGPT, large language models, code generation, non-determinism, reproducibility, temperature parameter, semantic similarity, syntactic similarity, structural similarity, HumanEval, APP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.