Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
Open accessThe DOI matches its Crossref record; 86 of the 88 references it cites were found in a registry or an index.
Abstract
Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT’s answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A recent study indicates that human users often prefer AI-generated content for its comprehensiveness and well-articulated language style, despite its inaccuracies [101].”
From Promises and challenges of generative artificial intelligence for human learning (2024) · cites this paper as [101]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Unless explicitly instructed otherwise, GPT-3.5 has a tendency to be verbose [Kabir et al. 2024] and confident [Hellas et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kabir et al. 2024
The cited source says what the claim attributes to it.
In the citing paper
“Still some testers like its comprehensiveness and good style of language presentation.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Kabir et al. (2024)
The cited source says what the claim attributes to it.
In the citing paper
“The main drawbacks of the LLM answers are fake information and excessive length of the content.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Kabir et al. (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Mixed Methods Research: A Research Paradigm Whose Time Has Come
Johnson, R. B., & Onwuegbuzie, A. J. (2004) · Educational Researcher · doi:10.3102/0013189x033007014 · Cited by 11,170 · cited 1×
“This work aims to bridge the gap by adopting a mixed-methods research design [48] with a combination of manual analysis, linguistic analysis, and user studies to compare human answers and ChatGPT answers to programming questions on Stack Overflow (SO).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Instead, we used Fleiss’s Kappa [32] score.”
Guest, G., MacQueen, K., & Namey, E. (2012) · doi:10.4135/9781483384436 · Cited by 7,924 · cited 2×
“The two authors met to review their labeling notes and performed thematic analysis [16, 40] to categorize the labels into four themes— Correctness , Consistency , Comprehensive-ness , Conciseness .”
BackgroundMethodOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 2×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundLikert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“Then, the participants started the study tasks by reading each SO question and rating their familiarity with the topic asked in the question (5-point Likert Scale [60]).”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 2×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
BackgroundSelf-citationIntroduction to Qualitative Research
Cirgin Ellett, M. L., & Beausang, C. C. (2002) · Gastroenterology Nursing · doi:10.1097/00001610-200201000-00004 · Cited by 1,306
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
BackgroundCOVID-19–Related Infodemic and Its Impact on Public Health: A Global Social Media Analysis
Islam, M. S., Sarkar, T., Khan, S. H., Mostofa Kamal, A.-H., Hasan, S. M. M., Kabir, A., Yeasmin, D., Islam, M. A., Amin Chowdhury, K. I., Anwar, K. S., et al. (2020) · The American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.20-0812 · Cited by 1,178 · cited 1×
“These types of misinformation have the power to mislead and mis-guide people and can potentially impair the normal functionalities of society and cause chaos [33, 45, 49, 79].”
BackgroundThe Emergence of Deepfake Technology: A Review
Westerlund, M. (2019) · Technology Innovation Management Review · doi:10.22215/timreview/1282 · Cited by 1,138 · cited 1×
“Previous studies have shown that content generated by LLM s may contain hallucinations and misinformation [14, 49, 85].”
BackgroundGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 1×
“Bender and Koll-er [10] show that any LLM s trained only on the form of language can not fully reach the human level of understanding.”
BackgroundA review: Knowledge reasoning over knowledge graph
Chen, X., Jia, S., & Xiang, Y. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.112948 · Cited by 1,035 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundTruth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking
Rashkin, H., Choi, E., Jang, J. Y., Volkova, S., & Choi, Y. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1317 · Cited by 870 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundDeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning
Xiong, W., Hoang, T., & Wang, W. Y. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1060 · Cited by 812 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundSoftware Psychology: Human Factors in Computer and Information Systems
Shneiderman, B. (1980) · CERN Document Server (European Organization for Nuclear Research) · Cited by 779
ChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · cited 3×
“Like other LLM s, ChatGPT is alsoplagued with these issues [15, 41, 50, 58].”
BackgroundSummarizing Source Code using a Neural Attention Model
Iyer, S., Konstas, I., Cheung, A., & Zettlemoyer, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1195 · Cited by 701 · cited 1×
“We followed prior work [46, 51] and trained a Support Vector Machine (SVM) classifier to predict the type of a SO question based on the question title.”
MethodRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173951 · Cited by 691 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Self-citationDesign lessons from the fastest q&a site in the west
Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979366 · Cited by 553 · cited 1×
“Through a mixed-methods study, Mamykina et al. [55] show that the chance of getting quick answers from the SO community is high.”
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514 · cited 2×
“Like other LLM s, ChatGPT is alsoplagued with these issues [15, 41, 50, 58].”
BackgroundA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1×
“Developers often use Copi-lot to get quick code suggestions and only turn to web searches to access the documentation or verify the suggestions [7, 73, 82].”
BackgroundGitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · cited 1×
“Furthermore, programmers also need to debug and fix those errors and make other modifications in order to integrate generated code into their program context, which may, in turn, impair their productivity [82].”
BackgroundHow do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405 · cited 3×
“The proliferation of social media and Q&A platforms for programming have immensely shaped the online help-seeking behavior of programmers [1, 76, 81, 84].”
All the News That’s Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation
Kreps, S., McCain, R. M., & Brundage, M. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2020.37 · Cited by 360 · cited 1×
“Specifically, several studies have investigated the power of AI-generated texts in de-ceiving people [17, 54].”
BackgroundLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“For example, we found an instance where the SO question asked about differences between public, private, and protected access modifiers in Java.”
BackgroundHow Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318 · cited 4×
“Like other LLM s, ChatGPT is alsoplagued with these issues [15, 41, 50, 58].”
BackgroundInfluentialZhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288 · cited 4×
“Zhou et al. [89] further highlight the risks by showing how traditional misinformation detection and mitigation methods often fail to identify misinformation generated by state-of-the-art LLM s.”
BackgroundMethodInfluentialWhat makes a good code example?: A study of programming Q&A in StackOverflow
Nasehi, S. M., Sillito, J., Maurer, F., & Burns, C. (2012) · 2012 28th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2012.6405249 · Cited by 287 · cited 1×
“Prior work [4, 8, 19, 59] shows that SO users preferred posts that contain illustrations, step-by-step instructions, multiple solutions, and positive sentiments.”
BackgroundStackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge
Vasilescu, B., Filkov, V., & Serebrenik, A. (2013) · 2013 International Conference on Social Computing · doi:10.1109/socialcom.2013.35 · Cited by 277 · cited 3×
“Another concern is the presence of toxicity and negative sentiment in people’s answers and comments on Stack Overflow.”
BackgroundRoss, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258 · cited 1×
“Developers often use Copi-lot to get quick code suggestions and only turn to web searches to access the documentation or verify the suggestions [7, 73, 82].”
BackgroundAnswering questions about unanswered questions of Stack Overflow
Asaduzzaman, M., Mashiyat, A. S., Roy, C. K., & Schneider, K. A. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624015 · Cited by 220 · cited 2×
“Prior work [4, 8, 19, 59] shows that SO users preferred posts that contain illustrations, step-by-step instructions, multiple solutions, and positive sentiments.”
BackgroundMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · cited 1×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
The impact of social media on software engineering practices and tools
Storey, M.-A., Treude, C., van Deursen, A., & Cheng, L.-T. (2010) · Proceedings of the FSE/SDP workshop on Future of software engineering research · doi:10.1145/1882362.1882435 · Cited by 213 · cited 2×
“The proliferation of social media and Q&A platforms for programming have immensely shaped the online help-seeking behavior of programmers [1, 76, 81, 84].”
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209479 · Cited by 204
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204
The Internal State of an LLM Knows When It's Lying
Azaria, A., & Mitchell, T. (2023) · arXiv · doi:10.48550/arxiv.2304.13734 · Cited by 179 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundThe Internal State of an LLM Knows When It’s Lying
Azaria, A., & Mitchell, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.68 · Cited by 175 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundWhat do developers search for on the web?
Xia, X., Bao, L., Lo, D., Kochhar, P. S., Hassan, A. E., & Xing, Z. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9514-4 · Cited by 174 · cited 2×
“Programmers often resort to online resources for a variety of programming tasks, e.g., API learning, bug fixing, comprehension of code or concepts, etc. [70, 75, 86].”
BackgroundSentiment Polarity Detection for Software Development
Calefato, F., Lanubile, F., Maiorano, F., & Novielli, N. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9546-9 · Cited by 171
The spread of emotion via facebook
Kramer, A. D. I. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207787 · Cited by 165 · cited 1×
“LIWC has been used to study AI-generated misin-formation [89], emotional expressions in social media posts [53], the success of human answers [8], etc.”
Assessing The Factual Accuracy of Generated Text
Goodrich, B., Rao, V., Liu, P. J., & Saleh, M. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330955 · Cited by 162 · cited 1×
“Previous studies show that LLM s can acquire factually incorrect knowledge during training and propagate the incorrect knowledge to generated content [9, 33, 35, 39, 56].”
Background'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
(2018) · doi:10.31235/osf.io/9wqxr · Cited by 161
Taking Flight with Copilot: Early insights and opportunities of AI-powered pair-programming tools
Bird, C., Ford, D., Zimmermann, T., Forsgren, N., Kalliamvakou, E., Lowdermilk, T., & Gazit, I. (2022) · Queue · doi:10.1145/3582083 · Cited by 158
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12813 · Cited by 156 · cited 1×
“Althoughapproaches have been proposed to mitigate hallucinations from LLM s [28, 65], they are only applicable to fixing Factual errors.”
How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow
Calefato, F., Lanubile, F., & Novielli, N. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.10.009 · Cited by 150 · cited 1×
“In a follow-up study, they found that novices and student programmers often encounter arrogant and rude comments on Stack Overflow, which discourages them from posting questions [20].”
BackgroundWhy, when, and what: Analyzing Stack Overflow questions by topic, type, and code
Allamanis, M., & Sutton, C. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624004 · Cited by 142 · cited 2×
“Previous studies [3, 51] show that linguistic forms of human answers on Stack Overflow vary based on the types of programming questions.”
BackgroundMethodJiang, S., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274351 · Cited by 125 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundIs GitHub copilot a substitute for human pair-programming?: an empirical study
Imai, S. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3522684 · Cited by 116 · cited 1×
“The online help-seeking behavior of developers is also changing along with other behavior shifts.”
BackgroundOn the Personality Traits of StackOverflow Users
Bazelli, B., Hindle, A., & Stroulia, E. (2013) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2013.72 · Cited by 114 · cited 4×
“Prior work [8]”
BackgroundMethodInfluential"Did you miss my comment or what?": understanding toxicity in open source discussions
Miller, C., Cohen, S., Klug, D., Vasilescu, B., & KaUstner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510111 · Cited by 111
Human Factors Experiments in Designing Interactive Systems
Shneiderman (1979) · Computer · doi:10.1109/mc.1979.1658571 · Cited by 111
DeepPath: A Reinforcement Learning Method for Knowledge Graph ReasoningPreprint
Xiong, W., Hoang, T., & Wang, W. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.06690 · Cited by 109 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundSequence labeling with multiple annotators
Rodrigues, F., Pereira, F. C., & Ribeiro, B. (2013) · Machine Learning · doi:10.1007/s10994-013-5411-2 · Cited by 108 · cited 1×
“To assess the quality and correctness of ChatGPT answers (RQ1), we used a standard NLP data labeling process [72, 88] to label the ChatGPT answers at the sentence level.”
MethodResponsible AI Systems: Who are the Stakeholders?
Deshpande, A., & Sharp, H. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534187 · Cited by 103 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3702320 · Cited by 91
Gamage, D., Ghasiya, P., Bonagiri, V., Whiting, M. E., & Sasahara, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517446 · Cited by 79 · cited 2×
“These types of misinformation have the power to mislead and mis-guide people and can potentially impair the normal functionalities of society and cause chaos [33, 45, 49, 79].”
BackgroundThe Signals that Potential Contributors Look for When Choosing Open-source Projects
Qiu, H. S., Li, Y. L., Padala, S., Sarma, A., & Vasilescu, B. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359224 · Cited by 75
Mitrović, S., Andreoletti, D., & Ayoub, O. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13852 · Cited by 75 · cited 2×
“The prevalence of misinformation, which can easily mislead users, has prompted Stack Overflow to impose a ban on answers generated by ChatGPT [64].”
BackgroundWhat Do Developers Use the Crowd For? A Study Using Stack Overflow
Abdalkareem, R., Shihab, E., & Rilling, J. (2017) · IEEE Software · doi:10.1109/ms.2017.31 · Cited by 74 · cited 1×
“The proliferation of social media and Q&A platforms for programming have immensely shaped the online help-seeking behavior of programmers [1, 76, 81, 84].”
BackgroundTruth, Lies, and Automation: How Language Models Could Change Disinformation
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K. (2021) · doi:10.51593/2021ca003 · Cited by 74 · cited 1×
“Specifically, several studies have investigated the power of AI-generated texts in de-ceiving people [17, 54].”
BackgroundKnowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.146 · Cited by 71 · cited 2×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
BackgroundRanking crowd knowledge to assist software development
de Souza, L. B. L., Campos, E. C., & Maia, M. D. A. (2014) · International Conference on Program Comprehension · doi:10.1145/2597008.2597146 · Cited by 63 · cited 1×
“Third, for question types, based on the literature [3, 25, 52, 81], we focused on three common question types— Conceptual, How-to , and Debugging .”
MethodEvaluating how developers use general-purpose web-search for code retrieval
Rahman, M. M., Barson, J., Paul, S., Kayani, J., Lois, F. A., Quezada, S. F., Parnin, C., Stolee, K. T., & Ray, B. (2018) · Proceedings of the 15th International Conference on Mining Software Repositories · doi:10.1145/3196398.3196425 · Cited by 62 · cited 1×
“A vast majority of these help-seeking activities include frequent engagement with community Q&A platforms such as Stack Overflow (SO) [69, 70, 84, 86].”
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey
Huang, Y., Feng, X., Feng, X., & Qin, B. (2021) · arXiv · doi:10.48550/arxiv.2104.14839 · Cited by 62 · cited 1×
“Previous work[21, 43,50]has shown that LLM s such as ChatGPT are prone to hallucination and may generate content with low quality.”
BackgroundHarnessing Stack Overflow for the IDE
Bacchelli, A., Ponzanelli, L., & Lanza, M. (2012) · 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE) · doi:10.1109/rsse.2012.6233404 · Cited by 60 · cited 1×
“For instance, since Q&A platforms are not integrated with IDEs, developers have to constantly switch between their IDEs and Q&A platforms, which may interrupt developers’ workflow and impair their performance persistence [6, 76, 84].”
BackgroundCalibrating Factual Knowledge in Pretrained Language ModelsPreprint
Dong, Q., Dai, D., Song, Y., Xu, J., Sui, Z., & Li, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03329 · Cited by 53
Sentiment polarity detection for software development
Calefato, F., Lanubile, F., Maiorano, F., & Novielli, N. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3182519 · Cited by 51
Human-Centered Responsible Artificial Intelligence: Current & Future Trends
Tahaei, M., Constantinides, M., Quercia, D., Kennedy, S., Muller, M., Stumpf, S., Liao, Q. V., Baeza-Yates, R., Aroyo, L., Holbrook, J., et al. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3583178 · Cited by 46 · cited 1×
“Therefore, it gives an extremely verbose response that contains the definitions of encapsulation, inheritance, etc., which is not useful in terms of identifying the differences originally asked for.”
BackgroundMining Successful Answers in Stack Overflow
Calefato, F., Lanubile, F., Marasciulo, M. C., & Novielli, N. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.56 · Cited by 45 · cited 3×
“LIWC is a psycholinguistic database that provides a dictionary of validated psycholinguistic lexicons in pre-determined categories that are psychologically meaningful.”
BackgroundCalibrating Factual Knowledge in Pretrained Language Models
Dong, Q., Dai, D., Song, Y., Xu, J., Sui, Z., & Li, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.438 · Cited by 39
SOSum: a dataset of stack overflow post summaries
Kou, B., Di, Y., Chen, M., & Zhang, T. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528487 · Cited by 28 · cited 1×
“Third, for question types, based on the literature [3, 25, 52, 81], we focused on three common question types— Conceptual, How-to , and Debugging .”
MethodSelf-citationA Review of Dataset and Labeling Methods for Causality Extraction
Xu, J., Zuo, W., Liang, S., & Zuo, X. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.133 · Cited by 25 · cited 1×
“To assess the quality and correctness of ChatGPT answers (RQ1), we used a standard NLP data labeling process [72, 88] to label the ChatGPT answers at the sentence level.”
MethodAnalyzing Web Search Behavior for Software Engineering Tasks
Rao, N., Bansal, C., Zimmermann, T., Awadallah, A. H., & Nagappan, N. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378083 · Cited by 24 · cited 2×
“Programmers often resort to online resources for a variety of programming tasks, e.g., API learning, bug fixing, comprehension of code or concepts, etc. [70, 75, 86].”
BackgroundGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 1×
“Developers often use Copi-lot to get quick code suggestions and only turn to web searches to access the documentation or verify the suggestions [7, 73, 82].”
BackgroundGeneration Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2302.07248 · Cited by 23 · cited 1×
“However, recent studies [2, 83] also show that only rendering the confidence level is not sufficient to help programmers understand the uncertainty and risks in the generated code.”
BackgroundAnalysis of Novices' Web-Based Help-Seeking Behavior While Programming
Skripchuk, J., Bennett, N., Zhang, J., Li, E., & Price, T. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569852 · Cited by 22 · cited 1×
“Programmers often resort to online resources for a variety of programming tasks, e.g., API learning, bug fixing, comprehension of code or concepts, etc. [70, 75, 86].”
BackgroundAutomated Summarization of Stack Overflow Posts
Kou, B., Chen, M., & Zhang, T.-Y. (2023) · International Conference on Software Engineering · doi:10.1109/icse48619.2023.00158 · Cited by 20 · cited 2×
“Previous studies [3, 51] show that linguistic forms of human answers on Stack Overflow vary based on the types of programming questions.”
BackgroundMethodSelf-citationKnowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · arXiv · doi:10.48550/arxiv.2106.09231 · Cited by 10 · cited 2×
“Besides, LLM s often generate fabricated texts that mimic truthful information and are hard to recognize, especially for users who lack the expertise [14, 21, 29].”
BackgroundQuality Estimation & Interpretability for Code Translation
Agarwal, M., Talamadupula, K., Houde, S., Martinez, F., Muller, M., Richards, J., Ross, S., & Weisz, J. D. (2020) · arXiv · doi:10.48550/arxiv.2012.07581 · Cited by 3 · cited 1×
“However, recent studies [2, 83] also show that only rendering the confidence level is not sufficient to help programmers understand the uncertainty and risks in the generated code.”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · arXiv · doi:10.48550/arxiv.2009.11462 · cited 2×
“Some recent work has investigated LLM s’ capability to generate fake news, images, and videos [27, 35, 89] about a multitude of social phenomena such as politics, elections, people, disease, economics, etc.”
Background
Cited by
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499
“For instance, although GenAI can improve the efficiency of information process ing and retrieval, there is a risk of fluency bias, where learners may overestimate their understanding due to the ease of cognitive infor mation processing 102,103 .”
The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking
Vallor, S. (2024) · doi:10.1093/oso/9780197759066.001.0001 · Cited by 239
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“Expertise Q&A on cybersecurity domain Kabir et al. (2024) conducts an empirical study of Chat-GPT’s performance in answering Stack Overflow programming questions.”
Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education
Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., & Malan, D. J. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630938 · Cited by 185
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Metadata
- Authors
- Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T.
- Affiliations
- Purdue University West Lafayette (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-17
- OpenAlex
- W4396832043
- Semantic Scholar
- 522346ac832b1ce03d07a1ba4b98cf746a5dc929
- Cited by
- 160
- References
- 77
- Citations by year
- 2026: 40 · 2025: 63 · 2024: 48
- FWCI
- 56.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, Stack Overflow, programming questions, Q&A platforms, answer correctness, linguistic analysis, user preferences, misinformation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.