Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT is the most widely adopted LLM in code generation in the literature.
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
From An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · cited 1× · see citing works
“As programming generation and assistant tools, such as CodeBERT [8] and IntelliCode Compose [9], become more widely used, there has been an increased focus on investigating the usability and interactions between users and code generation tools [3], [4], [10], [11].”
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · see citing works
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 1× · see citing works
“As programming generation and assistant tools, such as CodeBERT [8] and IntelliCode Compose [9], become more widely used, there has been an increased focus on investigating the usability and interactions between users and code generation tools [3], [4], [10], [11].”
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
“We find that prompts with a Chain-of-Thought strategy leads to more non-determinism when temperature=0, while code candidates generated from prompts requesting simple and concise code are more stable.”
Can ChatGPT evaluate research quality?
Thelwall, M. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2024-0013 · Cited by 68 · see citing works
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60 · see citing works
Garg, A., Nisumba Soodhani, K., & Rajendran, R. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100380 · Cited by 49 · see citing works
Ouyang, F., Guo, M., Zhang, N., Bai, X., & Jiao, P. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3486749 · Cited by 39 · see citing works
Navigating software development in the ChatGPT and GitHub copilot era
France, S. L. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.05.009 · Cited by 38 · see citing works
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27 · see citing works
Garg, A., & Rajendran, R. (2024) · Proceedings of the 16th International Conference on Computer Supported Education · doi:10.5220/0012693000003693 · Cited by 7 · see citing works
“Another investigation used the flake8 tool to assess code generated by AI against the PEP8 coding style, revealing a minimal syntax error rate of 2.88% (Feng, Y. et al., 2023).”
InfluentialAn Empirical Study of the Non-determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2023) · arXiv · doi:10.48550/arxiv.2308.02828
- Authors
- Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H.
- Affiliations
- University of North Texas (US)
- Argonne National Laboratory (US)
- Dakota State University
- Venue
- 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 876-885
- ISSN
- 0730-3157 (ISSN-L)
- OpenAlex
- W4385477851
- Cited by
- 117
- References
- 6
- Citations by year
- 2026: 12 · 2025: 41 · 2024: 51 · 2023: 13
- FWCI
- 30.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Annual International Computer Software and Applications Conference (COMPSAC) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- ChatGPT, code generation, large language models, crowdsourcing, social data, performance evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
