How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
To ensure the quality of its shared knowledge, Stack Overflow encourages users to revise answers through a badge system, which is based on quantitative measures (e.g., a badge is awarded after revising more than 500 answers). Prior studies show that badges can positively steer the user behavior on Stack Overflow (e.g., increasing user participation). However, little is known whether revision-related badges have a negative impact on the quality of revisions since some studies show that certain users may game incentive systems to gain rewards. In this study, we analyze 3,871,966 revision records that are collected from 2,377,692 Stack Overflow answers. We find that: 1) Users performed a much larger than usual revisions on the badge-awarding days compared to normal days; 25% of the users did not make any more revisions once they received their first revision-related badge. 2) Performing more revisions than usual in a single day increased the likelihood of such revisions being rolled back (e.g., due to undesired or incorrect revisions). 3) Users were more likely to perform text and small revisions if they performed many revisions in a single day. Our findings are concurred by the Stack Overflow community, and they highlight the need for changes to the current badge system in order to provide a better balance between the quality and quantity of revisions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493
Dominance statistics: Ordinal analyses to answer ordinal questions.
Cliff, N. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.494 · Cited by 1,485
Finding high-quality content in social media
Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341557 · Cited by 1,230
Social coding in GitHub: transparency and collaboration in an open software repository
Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145396 · Cited by 999
Improvements on Cross-Validation: The .632+ Bootstrap Method
Efron, B., & Tibshirani, R. J. (1997) · Journal of the American Statistical Association · doi:10.2307/2965703 · Cited by 723
Harnessing the wisdom of crowds in wikipedia: quality through coordination
Kittur, A., & Kraut, R. E. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460572 · Cited by 680
Expectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650
Predictors of answer quality in online Q&A sites
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357191 · Cited by 412
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
Research works printed but not linked: 2
- boslaugh, 2008, Statistics in a Nutshell a Desktop Quick Reference
- 10.1037/0033-2909.114.3.494 · doi:10.1037/0033-2909.114.3.494 · Paper page
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundWhy Are AI Agent–Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study
Alam, K., Mondal, S., & Roy, B. (2026) · Proceedings of the 23rd International Conference on Mining Software Repositories · doi:10.1145/3793302.3793611 · Cited by 2
“From the 2,113 unmerged cases identified in RQ1, we randomly sample 326 PRs, which provides a 95% confidence level with a 5% margin of error [11, 18].”
Metadata
- Authors
- Wang, S., Chen, T.-H., & Hassan, A. E.
- Affiliations
- Queen's University (CA)
- Concordia University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2020-09-01
- Volume
- 46
- Issue
- 9
- Pages
- 1024-1038
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2897316866
- Cited by
- 50
- References
- 44
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3 · 2023: 5 · 2022: 7 · 2021: 17 · 2020: 7
- FWCI
- 11.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Communication › Wikis in Education and Collaboration
- Keywords
- Stack Overflow, gamification, incentive mechanisms, user behavior, knowledge sharing communities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow. https://app.cochat.ai/papers/10.1109/tse.2018.2874470
- The paper
- Wang, S., Chen, T.-H., & Hassan, A. E. (2020). How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2018.2874470
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.