An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Open accessThe DOI matches its Crossref record; 85 of the 85 references it cites were found in a registry or an index; 6 of the 45 sources its claims cite were checked for support.
Abstract
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; non-deterministically returning very different code for the same prompt. Such non-determinism affects the correctness and consistency of the generated code, undermines developers’ trust in LLMs, and yields low reproducibility in LLM-based papers. Nevertheless, there is no work investigating how serious this non-determinism threat is. To fill this gap, this article conducts an empirical study on the non-determinism of ChatGPT in code generation. We chose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems across three code generation benchmarks (i.e., CodeContests, APPS and HumanEval) with three aspects of code similarities: semantic similarity, syntactic similarity, and structural similarity. Our results reveal that ChatGPT exhibits a high degree of non-determinism under the default setting: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00% and 47.56% for three different code generation datasets (i.e., CodeContests, APPS and HumanEval), respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration ( temperature \(=\) 1). In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.
Overview
CoChat Summary
TL;DR
This paper empirically studies non-determinism in ChatGPT code generation across 829 problems from CodeContests, APPS, and HumanEval. It finds substantial semantic, syntactic, and structural variation across repeated identical prompts, and shows that temperature 0 does not guarantee determinism.
Key findings
- Under default settings, the ratio of coding tasks with zero equal test output among five requests was 75.76% for CodeContests, 51.00% for APPS, and 47.56% for HumanEval; maximum test-pass-rate difference reached 1.00 on all three datasets and accounted for 39.63% of HumanEval problems.
- Setting temperature to 0 reduced but did not eliminate non-determinism: there were still 43.64% (CodeContests), 27.40% (APPS), and 18.29% (HumanEval) of problems with no equal test output among five candidates.
- GPT-4 was slightly more deterministic than GPT-3.5 at temperature 1, but their non-determinism was similar at temperature 0; Chain-of-Thought prompting led to more non-determinism at temperature 0, while prompts requesting concise code were more stable.
- In a manual review of 76 LLM-based code generation papers from the last two years, only 21.1% (16/76) considered non-determinism in their experimental evaluation.
Methods
- Empirical study of 829 code generation problems: CodeContests (165), APPS (500 randomly sampled), and HumanEval (164), using ChatGPT API models `gpt-3.5-turbo-0125` and `gpt-4-0613` with zero-shot Python3 prompts.
- For each task, the API was called five times with the same prompt; generated code was extracted, compiled, and run against benchmark test suites with a 3-second timeout.
- Non-determinism was measured via semantic similarity (test pass rate, output equivalence rate), syntactic similarity (Longest Common Subsequence, Levenshtein Edit Distance), and structural similarity (AST-based `pycode_similar` with Unified_Diff and Tree_Diff), with Kruskal-Wallis tests for significance.
Limitations
- The paper states that it does not study different programming languages, uses only a few code similarity measures, does not fully consider prompt influence on non-determinism, and focuses exclusively on ChatGPT rather than other LLMs.
- It also states that measuring semantic similarity through test execution outputs may oversimplify semantic similarity by ignoring code meaning, logic, and diverse correct solutions.
- The text makes evident that several analyses (RQ2, RQ3, RQ5, RQ6) report main results only for CodeContests, with other dataset results placed on the homepage, limiting the in-paper generalizability of those comparisons.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 2 Not valid · 18 Not checked · 125 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
What the sources say
- Partly supportedcites [15] [15] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. arXiv:2310.03533 [cs.SE] https: //arxiv.org/abs/2310.03533 - sourceExperimental — accuracy not yet measured
Our copy treats ChatGPT as a dominant subject of code-generation research — the background section ("A Large Language Model (LLM) refers to...", p. n/a) says the step change in GPT-3/3.5 performance attracted "a great deal of interest in GPT (and ChatGPT) in particular", and Section D (Scientific Evaluation of LLM-based Code Generation) surveys ChatGPT-centred work (Borji's failure analysis, Ouyang et al.'s non-determinism study, Yetistiren et al.'s comparison). However, the paper never states the superlative the claim attributes to it: it reports no counts or ranking of how often each LLM is used across the code-generation literature, and the only adoption figure it gives (Section C, "46% of the developers' code was written by Copilot", i.e. Codex/GPT-4-based Copilot in industry) is about practitioner use, not literature uptake. So the "widely used/much-studied" part is supported while "the most widely adopted" is not established by this source. (LLM, conf=0.68)
- Not checked yetcites [16] [16] Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) . IEEE Computer Society, Los Alamitos, CA, USA, 876-885. https://doi.org/10.1109/COMPSAC57700.2023.00117 - source
- Partly supportedcites [23] [23] Qi Guo, Junming Cao, Xiaofei Xie, Shangqing Liu, Xiaohong Li, Bihuan Chen, and Xin Peng. 2023. Exploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study. arXiv:2309.08221 [cs.SE] https: //arxiv.org/abs/2309.08221 - sourceExperimental — accuracy not yet measured
Section 2.2 identifies ChatGPT as an LLM and describes this paper's use of it for code refinement. Related Work says ChatGPT has been evaluated on several software-engineering tasks. Neither passage establishes that it is the most widely adopted LLM in the code-generation literature. (LLM, conf=0.83)
- Partly supportedcites [40] [40] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE] https://arxiv.org/abs/2305.01210 - sourceExperimental — accuracy not yet measured
The Additional information section includes ChatGPT among 26 models described as representative and popular for evaluation, and the paper evaluates its code generation. It does not compare adoption across the literature or establish that ChatGPT is the most widely adopted LLM for code generation. (LLM, conf=0.84)
- Partly supportedcites [42] [42] Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1-26. - sourceExperimental — accuracy not yet measured
Section 8.1 says ChatGPT has gained popularity for code generation, and the Introduction describes its rapid growth in users. Neither passage establishes that it is the most widely adopted LLM for code generation in the literature; the cited paper provides no comparison of adoption across models or studies. (LLM, conf=0.85)
- Partly supportedcites [63] [63] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Trans. Softw. Eng. 50, 4 (Feb. 2024), 911-936. https: //doi.org/10.1109/TSE.2024.3368208 - sourceExperimental — accuracy not yet measured
Section 5.1 LLM Models states that 'the most commonly utilized LLM in software testing tasks is ChatGPT' and that it 'ranks in the top position in terms of our collected studies' (102 papers), so the source does support ChatGPT being the most adopted LLM in the software-testing literature it surveyed. However, the claim's qualifier 'in code generation' is not what the source reports: the Introduction explicitly separates the two domains ('software testing possesses unique characteristics that differentiate it from code generation'), and Section 7 credits the NL2Code/code-generation survey to Zan et al. [168], a different work. The ranking is also bounded to the authors' collected studies rather than the whole literature. (LLM, conf=0.78)
- Not checked yetcites [70] [70] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE '24) . Association for Computing Machinery, New York, NY, USA, Article 37, 12 pages. https://doi.org/10.1145/3597503.3623316 - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Moreover, non-determinism affects the reliability and reproducibility of empirical software engineering [53].”
What the sources say
- Could not verifycites [53] [53] Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv:2201.11227 [cs.LG] https: //arxiv.org/abs/2201.11227 - sourceExperimental — accuracy not yet measured
Our copy of Synchromesh (Poesia et al., 2022) is about the reliability of LLM code generation — i.e., preventing syntactic/semantic implementation errors via Target Similarity Tuning and Constrained Semantic Decoding (Abstract p.1; Intro p.1-2; Discussion p.8-9) — not about non-determinism undermining empirical software engineering. The nearest content is its use of stochastic sampling (temperature τ = 0.7, Valid@K, Results p.8) and the observation that LLMs are sensitive to which few-shot examples are in the prompt (Intro p.2), but no passage frames non-determinism as a threat to reliability or reproducibility of empirical software engineering. Searched: 'non-determinism / reproducibility / empirical software engineering', 'stochastic sampling temperature reproducibility of experiments', 'deterministic decoding / outputs vary across runs', 'validity of experiments'; the copy is not truncated, so the point is simply absent rather than missing. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for three different code generation datasets (i.e., CodeContests, APPS, and HumanEval), respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we collect 76 LLM-based code generation papers that appeared in the last 2 years. Our manual analysis results highlight that only 21.1% of these papers consider the non-determinism threat in their experiments.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“accounts for 39.63% of the problems in HumanEval, the most widely used code generation benchmark.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
80 more claims
Create a free account to see all 146 claims and to verify the 38 references still open.
- It may state two separate claims as one. 38
- The quote only partly states this claim. 2
- Probably not a substantial claim. 33
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“One major issue is the inherent stochasticity in the models’ architecture, particularly in transformer-based models, which utilize probabilistic techniques during inference (e.g., beam search or temperature sampling)125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, Ouyang et al. reported that only a minority of studies take non-determinism into account in their experimental evaluation when using ChatGPT for code generation, suggesting that this limitation is also prevalent and overlooked in other domains of LLM use125.”
From Current applications and challenges in large language models for patient care: a systematic review (2025) · cites this paper as [125]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent work has identified multiple sources of non-determinism in large language models.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
The cited source says what the claim attributes to it.
In the citing paper
“For code generation specifically, Ouyang et al. (2025) find that repeated queries yield different implementations even with greedy sampling.”
From On Randomness in Agentic Evals (2026) · cites this paper as Ouyang et al. (2025)
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.2 RQ2: Influence of Temperature · p. 13
“When temperature =0.5, we observe that ChatGPT tends to generate code candidates that are more deterministic than temperature=1, but less deterministic than temperature=0.”
§ 4.2 RQ2: Influence of Temperature · p. 13
“| 0 | 0.41 | 0.39 | 0.67 | 0.50 | 0.46 | 0.74 | | 0.5 | 0.61 | 0.49 | 0.63 | 0.69 | 0.58 | 0.71 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 1 Introduction · p. 2
“the ratio of problems with not a single equal test output among the top-five code candidates is above 50% for all the benchmarks we study”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the ratios of tasks with zero equal test output (i.e., OER=0) among the five code candidates are 75.76%, 51.00%, and 47.56% for the three datasets, respectively.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“For temperature=1, we can observe that GPT-4 is slightly more deterministic than GPT-3.5, with lower test pass rate variance, lower ratio of worst cases, lower OER and OER (no ex.), lower LCS, higher LED, and lower structural similarity under two settings.”
§ 4.5 RQ5: GPT-4 vs. GPT-3.5 · p. 18
“| GPT-4 (t=1) | 0.78 | 0.68 | 0.79 | 0.82 | 0.74 | 0.84 | | GPT-3.5 (t=1) | 0.33 | 0.27 | 0.46 | 0.41 | 0.33 | 0.56 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“those generated from the Concise prompt have a high mean LCS value (0.07) and low LED value (11.77).”
§ 4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism · p. 19
“| Concise (t=0) | 0.70 | 0.53 | 0.71 | 11.77 | 20.55 | 12.14 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 RQ4: Coding Tasks Features and Non-determinism Degree · p. 15
“| R1 (t=0) | 0.18 | 0.00 | 0.00 | 0.00 | 1.20% | |”
§ 4.1 RQ1: Non-determinism of ChatGPT with Three Types of Similarities under default setting · p. 8
“the 'Ratio of worst cases', which is the ratio of problems with maximum diff of test pass rate being 1 or OER being 0.”
3 more findings
Create a free account to see every finding of this check.
Citations
Cites
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 5×
“It is one of the most widely used measurement metrics for assessing code generation capabilities 9 [4, 12, 26, 36, 71].”
InfluentialSummary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
Self-citationThe Corsini encyclopedia of psychology
(2010) · Choice Reviews Online · doi:10.5860/choice.47-6008 · Cited by 1,082
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 4×
“In our experiment, following the assessment practice of AlphaCode, we use the test set of CodeContests to benchmark the code generation tasks of ChatGPT.”
InfluentialCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
Self-citationCode as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · cited 2×
“Section 8 concludes.”
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07753 · Cited by 767 · cited 2×
“Section 8 concludes.”
Language to Logical Form with Neural Attention
Dong, L., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1004 · Cited by 700 · cited 1×
“To reduce the threat of prompt design, we use the most typical prompts that are the most widely used in LLM-based code generation and design an RQ to study their influence on non-determinism.”
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Self-citationA Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · arXiv · doi:10.48550/arxiv.1704.01696 · Cited by 585
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573 · cited 1×
“Section 2 introduces the main procedure of our study.”
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 1×
“The result shows that our measurements show similar evaluation results, which supports the robustness of our chosen measurements.”
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Self-citationAligning AI With Shared Human Values
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2008.02275 · Cited by 517
GraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480 · cited 1×
“When setting temperature=0, the code candidates generated from the Concise prompt are more deterministic than our Base prompt, while those code candidates generated from the CoT prompt suffer from higher randomness than our Base prompt.”
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1×
“As mentioned in Section 2, we especially focus on the code generated with Python3 language, since it is one of the most widely studied programming languages in code generation [4, 12, 17, 36, 59, 61, 64].”
BackgroundGreen, C. (1981) · Readings in Artificial Intelligence · doi:10.1016/b978-0-934613-03-3.50019-2 · Cited by 456 · cited 1×
“When setting temperature=0, the code candidates generated from the Concise prompt are more deterministic than our Base prompt, while those code candidates generated from the CoT prompt suffer from higher randomness than our Base prompt.”
Hassani, H., & Silva, E. S. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7020062 · Cited by 401 · cited 1×
“Firstly, ChatGPT has the ability to generate highly coherent and contextually appropriate responses to a wide variety of textual prompts [25].”
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390 · cited 2×
“Section 2 introduces the main procedure of our study.”
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 3×
“In this paper, we follow the common practice in LLM-based code generation assessment [4, 15].”
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385
Use chat gpt to solve programming bugs
Surameery, N. M. S., & Shakor, M. Y. (2023) · International Journal of Information technology and Computer Engineering · doi:10.55529/ijitc.31.17.22 · Cited by 382
Lee, M., Liang, P., & Yang, Q. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502030 · Cited by 345
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
Gulwani, S., Polozov, O., & Singh, R. (2017) · Foundations and Trends in Programming Languages · doi:10.1561/2500000010 · Cited by 319
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, C., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., & Berner, J. (2023) · neural information processing systems · doi:10.52202/075280-1205 · Cited by 312
Latent Predictor Networks for Code Generation
Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., & Senior, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1057 · Cited by 307 · cited 1×
“The threats to external validity mainly lie in the datasets, GPT versions, and prompt design in our study.”
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285 · cited 1×
“Relatively early work includes deductive synthesis approaches [19, 45] and inductive synthesis approaches [8, 56–58].”
MethodChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · cited 3×
“It potentially oversimplifies the multifaceted nature of semantic similarity, which should ideally encapsulate the code’s meaning and functionality rather than merely its output.”
Readings in Artificial Intelligence
(1981) · Elsevier eBooks · doi:10.1016/c2013-0-07694-3 · Cited by 239
A Methodology for LISP Program Construction from Examples
Summers, P. D. (1977) · Journal of the ACM · doi:10.1145/321992.322002 · Cited by 219 · cited 2×
“While the inductive synthesis approach was based on inductive specifications such as input/output pairs and examples etc, such as works on Lisp programs [8, 56, 58], Pygmalion [57] and more recently FlashFill [20].”
MethodTRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation
Yin, P., & Neubig, G. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2002 · Cited by 203 · cited 1×
“Another primary concern highlighted in our analysis revolves around the operationalization of semantic, syntactic, and structural similarities into measurable metrics for assessing code similarity.”
PYGMALION: A Creative Programming Environment
Smith, D. C. (1975) · doi:10.21236/ada016811 · Cited by 197 · cited 2×
“In the following, we use ‘Concise prompt’ to refer to the first prompt engineering strategy, and use ‘CoT prompt’ to refer to the second one for short.”
CERT: Continual Pre-Training on Sketches for Library-Oriented Code GenerationPreprint
Zan, D.-G., Chen, B., Yang, D.-J., Lin, Z., Kim, M., Guan, B., Wang, Y.-J., Chen, W.-Z., & Lou, J.-G. (2022) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2206.06888 · Cited by 172 · cited 2×
“Firstly, our analysis does not extend to the impact that different programming languages might have on the non-determinism of code generation.”
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151 · cited 1×
“The randomness in LLMs typically comes from the sampling methods used during text generation, such as top-k sampling or nucleus sampling [31, 49].”
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149 · cited 1×
“Section 2 introduces the main procedure of our study.”
Measuring Coding Challenge Competence With APPS
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.09938 · Cited by 142 · cited 2×
“Our experiments use the three most widely studied code generation benchmarks: CodeContest [36], APPS [26], and HumanEval [12].”
CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., & Xie, T. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623316 · Cited by 140
Self-citationExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot
Mastropaolo, A., Pascarella, L., Guglielmi, E., Ciniselli, M., Scalabrino, S., Oliveto, R., & Bavota, G. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00181 · Cited by 138 · cited 1×
“We choose the Longest Common Subsequence and Levenshtein Edit Distance as evaluation tools [34, 35, 46, 66].”
A Grammar-Based Structural CNN Decoder for Code GenerationPreprint
Sun, Z.-Y., Zhu, Q.-H., Mou, L.-L., Xiong, Y., Li, G., & Zhang, L. (2018) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.1811.06837 · Cited by 138 · cited 2×
“Instead of RNN, Sun et al. [59] propose a grammar-based structural CNN to capture the long dependency in code.”
BackgroundExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623306 · Cited by 136
The Inference of Regular LISP Programs from Examples
Biermann, A. W. (1978) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/tsmc.1978.4310035 · Cited by 118 · cited 1×
“When setting temperature=0, the code candidates generated from the Concise prompt are more deterministic than our Base prompt, while those code candidates generated from the CoT prompt suffer from higher randomness than our Base prompt.”
Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117 · cited 2×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
A Grammar-Based Structural CNN Decoder for Code Generation
Sun, Z., Zhu, Q., Mou, L., Xiong, Y., Li, G., & Zhang, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017055 · Cited by 107
Proceedings of the 28th International Conference on Program Comprehension
(2020) · doi:10.1145/3387904 · Cited by 104
SPoC: Search-based Pseudocode to Code
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., & Liang, P. (2019) · arXiv · doi:10.48550/arxiv.1906.04908 · Cited by 103 · cited 1×
“Some papers use the pass@k metric [12, 32] (i.e., the ratio of coding tasks with 100% test pass rate) to indicate the high-level code generation correctness of a code generation approach.”
Problems and opportunities in training deep learning software systems: an analysis of variance
Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416545 · Cited by 100
RankGen: Improving Text Generation with Large Ranking ModelsPreprint
Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.09726 · Cited by 94 · cited 1×
“The randomness in LLMs typically comes from the sampling methods used during text generation, such as top-k sampling or nucleus sampling [31, 49].”
Latent Predictor Networks for Code Generation
Ling, W., Grefenstette, E., Hermann, K. M., Kočiský, T., Senior, A., Wang, F., & Blunsom, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06744 · Cited by 83 · cited 1×
“The threats to external validity mainly lie in the datasets, GPT versions, and prompt design in our study.”
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 78 · cited 1×
“This non-determinism (i.e., the inconsistency in the code candidates generated in different requests with identical prompts) 1 is an essential consideration when using LLM in practice [5].”
Yetiştiren, B., Özsoy, I., Ayerdem, M., & Tüzün, E. (2023) · arXiv · doi:10.48550/arxiv.2304.10778 · Cited by 75 · cited 1×
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Synchromesh: Reliable code generation from pre-trained language models
Poesia, G., Polozov, O., Le, V., Tiwari, A., Soares, G., Meek, C., & Gulwani, S. (2022) · arXiv · doi:10.48550/arxiv.2201.11227 · Cited by 72
CodeEditor : Learning to Edit Source Code with Pre-trained Models
Li, J., Li, G., Li, Z., Jin, Z. G., Hu, X., Zhang, K., & Fu, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3597207 · Cited by 69
Inferring LISP Programs from Examples
Shaw, D. E., Swartout, W., & Green, C. C. (1975) · Columbia Academic Commons (Columbia University) · doi:10.7916/d89k4k6x · Cited by 66
SkCoder: A Sketch-based Approach for Automatic Code Generation
Li, J., Li, Y., Li, G., Jin, Z., Hao, Y., & Hu, X. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00179 · Cited by 64
Fault-Aware Neural Code RankersPreprint
Inala, J., Wang, C., Yang, M., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.03865 · Cited by 62
Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTPreprint
Bhavya, B., Xiong, J.-J., & Zhai, C.-X. (2022) · Proceedings of the 15th International Conference on Natural Language Generation · doi:10.48550/arxiv.2210.04186 · Cited by 55 · cited 2×
“Section 8 concludes.”
Code Generation as a Dual Task of Code Summarization
Wei, B., Li, G., Xia, X., Fu, Z., & Jin, Z. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05923 · Cited by 53 · cited 1×
“Compared with the Base prompt and Concise prompt, the CoT prompt has a higher mean-variance (0.02), higher mean maximum difference (0.15), and a rather higher ratio of worst cases (1.82%).”
InfluentialCCTEST: Testing and Repairing Code Completion Systems
Li, Z., Wang, C., Liu, Z., Wang, H., Chen, D., Wang, S., & Gao, C. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00110 · Cited by 52 · cited 1×
“The test pass rate calculates the ratio of the passed test case number against the total test case number for code candidates.”
COMET: Coverage-guided Model Generation For Deep Learning Library Testing
Li, M., Cao, J., Tian, Y., Li, T. O., Wen, M., & Cheung, S.-C. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3583566 · Cited by 45
Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPT
Bhavya, B., Xiong, J., & Zhai, C. (2022) · Proceedings of the 15th International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.25 · Cited by 40
From Sparse to Soft Mixtures of Experts
Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023) · arXiv · doi:10.48550/arxiv.2308.00951 · Cited by 36 · cited 1×
“The reason for such non-determinism with the temperature being zero is still controversial [1], with different hypotheses such as floating point, unreliable GPU calculations, and its sparse MoE architecture failing to enforce per-sequence determinism [44, 54].”
Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
Nagarajan, P., Warnell, G., & Stone, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05676 · Cited by 33 · cited 1×
“We choose n =5, since 5 is a widely used figure in the papers studying variance [50]. n =5 is only used in RQ3.”
RankGen: Improving Text Generation with Large Ranking Models
Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.15 · Cited by 30
Chatterjee, S., Saha, D., Sharma, A., & Verma, Y. (2022) · Annals of Operations Research · doi:10.1007/s10479-021-04258-y · Cited by 22
TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code GenerationPreprint
Yin, P., & Neubig, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02720 · Cited by 18 · cited 1×
“Another primary concern highlighted in our analysis revolves around the operationalization of semantic, syntactic, and structural similarities into measurable metrics for assessing code similarity.”
Fault-Aware Neural Code Rankers
Inala, J. P., Wang, C., Yang, M., Codas, A., Encarnación, M. J., Lahiri, S. K., Musuvathi, M., & Gao, J. (2022) · neural information processing systems · doi:10.52202/068431-0976 · Cited by 16
How are Deep Learning Models Similar? An Empirical Study on Clone Analysis of Deep Learning Software
Wu, X., Qin, L., Yu, B., Xie, X., Ma, L., Xue, Y., Liu, Y., & Zhao, J. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389254 · Cited by 7 · cited 1×
“We use a Python library called pycode_similar 10 [37, 65] to calculate the similarity.”
On the Robustness of Code Generation Techniques: An Empirical Study on GitHub CopilotPreprint
Mastropaolo, A., Pascarella, L., Guglielmi, E., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00438 · Cited by 7 · cited 1×
“We choose the Longest Common Subsequence and Levenshtein Edit Distance as evaluation tools [34, 35, 46, 66].”
SkCoder: A Sketch-based Approach for Automatic Code GenerationPreprint
Li, J., Li, Y., Li, G., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.06144 · Cited by 5 · cited 1×
“ChatGPT has gained widespread popularity and recognition in multiple tasks including question-answering, language translation, sentiment analysis, and text summarising, among which code generation is one of the most impressive tasks [10, 36].”
CCTEST: Testing and Repairing Code Completion SystemsPreprint
Li, Z., Wang, C., Liu, Z., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.08289 · Cited by 2 · cited 1×
“The test pass rate calculates the ratio of the passed test case number against the total test case number for code candidates.”
Language Models as a Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A. G., Shadbolt, N., & Wooldridge, M. (2023) · arXiv · doi:10.48550/arxiv.2309.16573 · Cited by 1 · cited 1×
“The reason for such non-determinism with the temperature being zero is still controversial [1], with different hypotheses such as floating point, unreliable GPU calculations, and its sparse MoE architecture failing to enforce per-sequence determinism [44, 54].”
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Research works printed but not linked: 7
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
- Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
- Andrej Kiviriga. 2023. Efficient Model Checking: The Power of Randomness. Aalborg Universitetsforlag.
- Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems.
- Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. In The Corsini Encyclopedia of Psychology. John Wiley & Sons, Inc., 1–1.
- OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. Technical Report, OpenAI.
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“LLMs are inherently nondeterministic: the same prompt produces different answers on different inference executions (unless the temperature is set to zero, which has often been found to be suboptimal over multiple executions) [9].”
BackgroundInfluentialSelf-citationLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Self-citationCurrent applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“Error analysis [31] Mistakes made during task performance are analyzed to understand metacognitive flexibility.”
BackgroundKernberg, A., Gold, J. A., & Mohan, V. (2024) · Journal of Medical Internet Research · doi:10.2196/54419 · Cited by 120
Metadata
- Authors
- Ouyang, S., Zhang, J. M., Harman, M., & Wang, M.
- Affiliations
- University of London (GB)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-01-22
- Volume
- 34
- Issue
- 2
- Pages
- 1-28
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4402860127
- Semantic Scholar
- dc7de606ecf7c65abc26396629b633e18d8704d5
- Cited by
- 258
- References
- 41
- Citations by year
- 2026: 108 · 2025: 90
- FWCI
- 94.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Explore Bristol Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- UCL Discovery (University College London) (submittedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by)
- Bristol Research (University of Bristol) (submittedVersion, cc-by)
- arXiv (Cornell University) (public-domain)
- Funders
- UKRI Centre for Doctoral Training in Safe and Trusted Artificial Intelligence (EP/S023356/1)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- ChatGPT, large language models, code generation, non-determinism, reproducibility, temperature parameter, semantic similarity, syntactic similarity, structural similarity, HumanEval, APP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.