CoChat

An Empirical Study of the Non-Determinism of ChatGPT in Code Generation

Open access
Ouyang, Shuyin, Zhang, Jie M., Harman, Mark, Wang, Meng · 2025 · ACM Transactions on Software Engineering and Methodology

The DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 6 of the 45 sources its claims cite were checked for support.

Abstract

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Overview

CoChat Summary

TL;DR

This paper empirically studies non-determinism in ChatGPT code generation across 829 problems from CodeContests, APPS, and HumanEval. It finds substantial semantic, syntactic, and structural variation across repeated identical prompts, and shows that temperature 0 does not guarantee determinism.

Key findings

  • Under default settings, the ratio of coding tasks with zero equal test output among five requests was 75.76% for CodeContests, 51.00% for APPS, and 47.56% for HumanEval; maximum test-pass-rate difference reached 1.00 on all three datasets and accounted for 39.63% of HumanEval problems.
  • Setting temperature to 0 reduced but did not eliminate non-determinism: there were still 43.64% (CodeContests), 27.40% (APPS), and 18.29% (HumanEval) of problems with no equal test output among five candidates.
  • GPT-4 was slightly more deterministic than GPT-3.5 at temperature 1, but their non-determinism was similar at temperature 0; Chain-of-Thought prompting led to more non-determinism at temperature 0, while prompts requesting concise code were more stable.
  • In a manual review of 76 LLM-based code generation papers from the last two years, only 21.1% (16/76) considered non-determinism in their experimental evaluation.

Methods

  • Empirical study of 829 code generation problems: CodeContests (165), APPS (500 randomly sampled), and HumanEval (164), using ChatGPT API models `gpt-3.5-turbo-0125` and `gpt-4-0613` with zero-shot Python3 prompts.
  • For each task, the API was called five times with the same prompt; generated code was extracted, compiled, and run against benchmark test suites with a 3-second timeout.
  • Non-determinism was measured via semantic similarity (test pass rate, output equivalence rate), syntactic similarity (Longest Common Subsequence, Levenshtein Edit Distance), and structural similarity (AST-based `pycode_similar` with Unified_Diff and Tree_Diff), with Kruskal-Wallis tests for significance.

Limitations

  • The paper states that it does not study different programming languages, uses only a few code similarity measures, does not fully consider prompt influence on non-determinism, and focuses exclusively on ChatGPT rather than other LLMs.
  • It also states that measuring semantic similarity through test execution outputs may oversimplify semantic similarity by ignoring code meaning, logic, and diverse correct solutions.
  • The text makes evident that several analyses (RQ2, RQ3, RQ5, RQ6) report main results only for CodeContests, with other dataset results placed on the homepage, limiting the in-paper generalizability of those comparisons.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3697010

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 2 Not valid · 18 Not checked · 125 Own results and interpretations

80 more claims

Create a free account to see all 146 claims and to verify the 38 references still open.

See all claims — free account
  • It may state two separate claims as one. 38
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 33

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 8 · the rest aren’t loaded

3 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Research works printed but not linked: 7

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
  2. Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
  3. Andrej Kiviriga. 2023. Efficient Model Checking: The Power of Randomness. Aalborg Universitetsforlag.
  4. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems.
  5. Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. In The Corsini Encyclopedia of Psychology. John Wiley & Sons, Inc., 1–1.
  6. OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. · arXiv
  7. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Technical Report, OpenAI.

Cited by

See all 262

Metadata

Authors
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M.
Affiliations
University of London (GB)
Venue
ACM Transactions on Software Engineering and Methodology
Published
2025-01-22
Volume
34
Issue
2
Pages
1-28
ISSN
  • 1049-331X (ISSN-L)
  • 1557-7392
OpenAlex
W4402860127
Cited by
258
References
41
Citations by year
2026: 108 · 2025: 90
FWCI
94.52
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
  • Explore Bristol Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • UCL Discovery (University College London) (submittedVersion)
  • Research Portal (King's College London) (submittedVersion, cc-by)
  • Bristol Research (University of Bristol) (submittedVersion, cc-by)
  • arXiv (Cornell University) (public-domain)
Funders
UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (EP/S023356/1)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
ChatGPT, large language models, code generation, non-determinism, reproducibility, temperature parameter, semantic similarity, syntactic similarity, structural similarity, HumanEval, APP

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.