An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 6 of the 45 sources its claims cite were checked for support.
Abstract
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.
Overview
CoChat Summary
TL;DR
This paper empirically studies non-determinism in ChatGPT code generation across 829 problems from CodeContests, APPS, and HumanEval. It finds substantial semantic, syntactic, and structural variation across repeated identical prompts, and shows that temperature 0 does not guarantee determinism.
Key findings
- Under default settings, the ratio of coding tasks with zero equal test output among five requests was 75.76% for CodeContests, 51.00% for APPS, and 47.56% for HumanEval; maximum test-pass-rate difference reached 1.00 on all three datasets and accounted for 39.63% of HumanEval problems.
- Setting temperature to 0 reduced but did not eliminate non-determinism: there were still 43.64% (CodeContests), 27.40% (APPS), and 18.29% (HumanEval) of problems with no equal test output among five candidates.
- GPT-4 was slightly more deterministic than GPT-3.5 at temperature 1, but their non-determinism was similar at temperature 0; Chain-of-Thought prompting led to more non-determinism at temperature 0, while prompts requesting concise code were more stable.
- In a manual review of 76 LLM-based code generation papers from the last two years, only 21.1% (16/76) considered non-determinism in their experimental evaluation.
Methods
- Empirical study of 829 code generation problems: CodeContests (165), APPS (500 randomly sampled), and HumanEval (164), using ChatGPT API models `gpt-3.5-turbo-0125` and `gpt-4-0613` with zero-shot Python3 prompts.
- For each task, the API was called five times with the same prompt; generated code was extracted, compiled, and run against benchmark test suites with a 3-second timeout.
- Non-determinism was measured via semantic similarity (test pass rate, output equivalence rate), syntactic similarity (Longest Common Subsequence, Levenshtein Edit Distance), and structural similarity (AST-based `pycode_similar` with Unified_Diff and Tree_Diff), with Kruskal-Wallis tests for significance.
Limitations
- The paper states that it does not study different programming languages, uses only a few code similarity measures, does not fully consider prompt influence on non-determinism, and focuses exclusively on ChatGPT rather than other LLMs.
- It also states that measuring semantic similarity through test execution outputs may oversimplify semantic similarity by ignoring code meaning, logic, and diverse correct solutions.
- The text makes evident that several analyses (RQ2, RQ3, RQ5, RQ6) report main results only for CodeContests, with other dataset results placed on the homepage, limiting the in-paper generalizability of those comparisons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 2 Not valid · 18 Not checked · 125 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
What the sources say
- Partly supportedcites [15] [15] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. arXiv:2310.03533 [cs.SE] https: //arxiv.org/abs/2310.03533 - sourceExperimental — accuracy not yet measured
Our copy treats ChatGPT as a dominant subject of code-generation research — the background section ("A Large Language Model (LLM) refers to...", p. n/a) says the step change in GPT-3/3.5 performance attracted "a great deal of interest in GPT (and ChatGPT) in particular", and Section D (Scientific Evaluation of LLM-based Code Generation) surveys ChatGPT-centred work (Borji's failure analysis, Ouyang et al.'s non-determinism study, Yetistiren et al.'s comparison). However, the paper never states the superlative the claim attributes to it: it reports no counts or ranking of how often each LLM is used across the code-generation literature, and the only adoption figure it gives (Section C, "46% of the developers' code was written by Copilot", i.e. Codex/GPT-4-based Copilot in industry) is about practitioner use, not literature uptake. So the "widely used/much-studied" part is supported while "the most widely adopted" is not established by this source. (LLM, conf=0.68)
- Not checked yetcites [16] [16] Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE Computer Society, Los Alamitos, CA, USA, 876-885. https://doi.org/10.1109/COMPSAC57700.2023.00117 - source
- Partly supportedcites [23] [23] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2023. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. arXiv:2309.08221 [cs.SE] https: //arxiv.org/abs/2309.08221 - sourceExperimental — accuracy not yet measured
Section 2.2 identifies ChatGPT as an LLM and describes this paper's use of it for code refinement. Related Work says ChatGPT has been evaluated on several software-engineering tasks. Neither passage establishes that it is the most widely adopted LLM in the code-generation literature. (LLM, conf=0.83)
- Partly supportedcites [40] [40] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 - sourceExperimental — accuracy not yet measured
The Additional information section includes ChatGPT among 26 models described as representative and popular for evaluation, and the paper evaluates its code generation. It does not compare adoption across the literature or establish that ChatGPT is the most widely adopted LLM for code generation. (LLM, conf=0.84)
- Partly supportedcites [42] [42] Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1-26. - sourceExperimental — accuracy not yet measured
Section 8.1 says ChatGPT has gained popularity for code generation, and the Introduction describes its rapid growth in users. Neither passage establishes that it is the most widely adopted LLM for code generation in the literature; the cited paper provides no comparison of adoption across models or studies. (LLM, conf=0.85)
- Partly supportedcites [63] [63] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Trans. Softw. Eng. 50, 4 (Feb. 2024), 911-936. https: //doi.org/10.1109/TSE.2024.3368208 - sourceExperimental — accuracy not yet measured
Section 5.1 LLM Models states that 'the most commonly utilized LLM in software testing tasks is ChatGPT' and that it 'ranks in the top position in terms of our collected studies' (102 papers), so the source does support ChatGPT being the most adopted LLM in the software-testing literature it surveyed. However, the claim's qualifier 'in code generation' is not what the source reports: the Introduction explicitly separates the two domains ('software testing possesses unique characteristics that differentiate it from code generation'), and Section 7 credits the NL2Code/code-generation survey to Zan et al. [168], a different work. The ranking is also bounded to the authors' collected studies rather than the whole literature. (LLM, conf=0.78)
- Not checked yetcites [70] [70] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24) . Association for Computing Machinery, New York, NY, USA, Article 37, 12 pages. https://doi.org/10.1145/3597503.3623316 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, non-determinism affects the reliability and reproducibility of empirical software engineering [53].”
What the sources say
- Could not verifycites [53] [53] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv:2201.11227 [cs.LG] https: //arxiv.org/abs/2201.11227 - sourceExperimental — accuracy not yet measured
Our copy of Synchromesh (Poesia et al., 2022) is about the reliability of LLM code generation — i.e., preventing syntactic/semantic implementation errors via Target Similarity Tuning and Constrained Semantic Decoding (Abstract p.1; Intro p.1-2; Discussion p.8-9) — not about non-determinism undermining empirical software engineering. The nearest content is its use of stochastic sampling (temperature τ = 0.7, Valid@K, Results p.8) and the observation that LLMs are sensitive to which few-shot examples are in the prompt (Intro p.2), but no passage frames non-determinism as a threat to reliability or reproducibility of empirical software engineering. Searched: 'non-determinism / reproducibility / empirical software engineering', 'stochastic sampling temperature reproducibility of experiments', 'deterministic decoding / outputs vary across runs', 'validity of experiments'; the copy is not truncated, so the point is simply absent rather than missing. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for three different code generation datasets (i.e., CodeContests, APPS, and HumanEval), respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we collect 76 LLM-based code generation papers that appeared in the last 2 years. Our manual analysis results highlight that only 21.1% of these papers consider the non-determinism threat in their experiments.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“accounts for 39.63% of the problems in HumanEval, the most widely used code generation benchmark.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
80 more claims
Create a free account to see all 146 claims and to verify the 38 references still open.
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
- Probably not a substantial claim. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One major issue is the inherent stochasticity in the models’ architecture, particularly in transformer-based models, which utilize probabilistic techniques during inference (e.g., beam search or temperature sampling)125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, Ouyang et al. reported that only a minority of studies take non-determinism into account in their experimental evaluation when using ChatGPT for code generation, suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent work has identified multiple sources of non-determinism in large language models.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
The cited source says what the claim attributes to it.
In the citing paper
“For code generation specifically, Ouyang et al. (2025) find that repeated queries yield different implementations even with greedy sampling.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.2 RQ2: Influence of Temperature · p. 13
“When temperature =0.5, we observe that ChatGPT tends to generate code candidates that are more deterministic than temperature=1, but less deterministic than temperature=0.”
§ 4.2 RQ2: Influence of Temperature · p. 13
“| 0 | 0.41 | 0.39 | 0.67 | 0.50 | 0.46 | 0.74 | | 0.5 | 0.61 | 0.49 | 0.63 | 0.69 | 0.58 | 0.71 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 1 Introduction · p. 2
“the ratio of problems with not a single equal test output among the top-five code candidates is above 50% for all the benchmarks we study”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the ratios of tasks with zero equal test output (i.e., OER=0) among the five code candidates are 75.76%, 51.00%, and 47.56% for the three datasets, respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“For temperature=1, we can observe that GPT-4 is slightly more deterministic than GPT-3.5, with lower test pass rate variance, lower ratio of worst cases, lower OER and OER (no ex.), lower LCS, higher LED, and lower structural similarity under two settings.”
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“| GPT-4 (t=1) | 0.78 | 0.68 | 0.79 | 0.82 | 0.74 | 0.84 | | GPT-3.5 (t=1) | 0.33 | 0.27 | 0.46 | 0.41 | 0.33 | 0.56 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“those generated from the Concise prompt have a high mean LCS value (0.07) and low LED value (11.77).”
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“| Concise (t=0) | 0.70 | 0.53 | 0.71 | 11.77 | 20.55 | 12.14 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 RQ4: Coding Tasks Features and Non-determinism Degree · p. 15
“| R1 (t=0) | 0.18 | 0.00 | 0.00 | 0.00 | 1.20% | |”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the 'Ratio of worst cases', which is the ratio of problems with maximum diff of test pass rate being 1 or OER being 0.”
3 more findings
Create a free account to see every finding of this check.
Citations
Cites
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 5×
“It is one of the most widely used measurement metrics for assessing code generation capabilities 9 [4, 12, 26, 36, 71].”
InfluentialSummary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Self-citationThe Corsini encyclopedia of psychology
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6008 · Cited by 1,082
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 4×
“In our experiment, following the assessment practice of AlphaCode, we use the test set of CodeContests to benchmark the code generation tasks of ChatGPT.”
InfluentialCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Self-citationCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · cited 2×
“Section 8 concludes.”
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07753 · Cited by 767 · cited 2×
“Section 8 concludes.”
Research works printed but not linked: 7
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
- Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
- Andrej Kiviriga. 2023. Efficient Model Checking: The Power of Randomness. Aalborg Universitetsforlag.
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems.
- Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. In The Corsini Encyclopedia of Psychology. John Wiley & Sons, Inc., 1–1.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Technical Report, OpenAI.
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“LLMs are inherently nondeterministic: the same prompt produces different answers on different inference executions (unless the temperature is set to zero, which has often been found to be suboptimal over multiple executions) [9].”
BackgroundInfluentialSelf-citationLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Self-citationCurrent applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“Error analysis [31] Mistakes made during task performance are analyzed to understand metacognitive flexibility.”
BackgroundKernberg, A., Gold, J. A., & Mohan, V. (2024) · Journal of Medical Internet Research · doi:10.2196/54419 · Cited by 120
Metadata
- Authors
- Ouyang, S., Zhang, J. M., Harman, M., & Wang, M.
- Affiliations
- University of London (GB)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-01-22
- Volume
- 34
- Issue
- 2
- Pages
- 1-28
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402860127
- Semantic Scholar
- dc7de606ecf7c65abc26396629b633e18d8704d5
- Cited by
- 258
- References
- 41
- Citations by year
- 2026: 108 · 2025: 90
- FWCI
- 94.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Explore Bristol Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- arXiv (Cornell University) (public-domain)
- Funders
- UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (EP/S023356/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- ChatGPT, large language models, code generation, non-determinism, reproducibility, temperature parameter, semantic similarity, syntactic similarity, structural similarity, HumanEval, APP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. https://app.cochat.ai/papers/10.1145/3697010
- The paper
- Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025). An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3697010
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.