Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
Open accessThe DOI matches its Crossref record; 88 of the 88 references it cites were found in a registry or an index.
Abstract
Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT’s answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A recent study indicates that human users often prefer AI-generated content for its comprehensiveness and well-articulated language style, despite its inaccuracies [101].”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [101]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Unless explicitly instructed otherwise, GPT-3.5 has a tendency to be verbose [Kabir et al. 2024] and confident [Hellas et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kabir et al. 2024
The cited source says what the claim attributes to it.
In the citing paper
“Still some testers like its comprehensiveness and good style of language presentation.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Kabir et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“The main drawbacks of the LLM answers are fake information and excessive length of the content.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Kabir et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Mixed Methods Research: A Research Paradigm Whose Time Has Come
Johnson, R. B., & Onwuegbuzie, A. J. (2004) · Educational Researcher · doi:10.3102/0013189x033007014 · Cited by 11,170 · cited 1×
“This work aims to bridge the gap by adopting a mixed-methods research design [48] with a combination of manual analysis, linguistic analysis, and user studies to compare human answers and ChatGPT answers to programming questions on Stack Overflow (SO).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Instead, we used Fleiss’s Kappa [32] score.”
Guest, G., MacQueen, K., & Namey, E. (2012) · doi:10.4135/9781483384436 · Cited by 7,924 · cited 2×
“The two authors met to review their labeling notes and performed thematic analysis [16, 40] to categorize the labels into four themes— Correctness , Consistency , Comprehensive-ness , Conciseness .”
BackgroundMethodOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundLikert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“Then, the participants started the study tasks by reading each SO question and rating their familiarity with the topic asked in the question (5-point Likert Scale [60]).”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 2×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
BackgroundSelf-citationIntroduction to Qualitative Research
Cirgin Ellett, M. L., & Beausang, C. C. (2002) · Gastroenterology Nursing · doi:10.1097/00001610-200201000-00004 · Cited by 1,306
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
Background
Cited by
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499
“For instance, although GenAI can improve the efficiency of information process ing and retrieval, there is a risk of fluency bias, where learners may overestimate their understanding due to the ease of cognitive infor mation processing 102,103 .”
The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking
Vallor, S. (2024) · doi:10.1093/oso/9780197759066.001.0001 · Cited by 239
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“Expertise Q&A on cybersecurity domain Kabir et al. (2024) conducts an empirical study of Chat-GPT’s performance in answering Stack Overflow programming questions.”
Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Metadata
- Authors
- Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T.
- Affiliations
- Purdue University West Lafayette (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-17
- OpenAlex
- W4396832043
- Semantic Scholar
- 522346ac832b1ce03d07a1ba4b98cf746a5dc929
- Cited by
- 160
- References
- 77
- Citations by year
- 2026: 40 · 2025: 63 · 2024: 48
- FWCI
- 56.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, Stack Overflow, programming questions, Q&A platforms, answer correctness, linguistic analysis, user preferences, misinformation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. https://app.cochat.ai/papers/10.1145/3613904.3642596
- The paper
- Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024). Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642596
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.