How Do Users Revise Answers on Technical Q&A Websites? A Case Study on Stack Overflow
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
To ensure the quality of its shared knowledge, Stack Overflow encourages users to revise answers through a badge system, which is based on quantitative measures (e.g., a badge is awarded after revising more than 500 answers). Prior studies show that badges can positively steer the user behavior on Stack Overflow (e.g., increasing user participation). However, little is known whether revision-related badges have a negative impact on the quality of revisions since some studies show that certain users may game incentive systems to gain rewards. In this study, we analyze 3,871,966 revision records that are collected from 2,377,692 Stack Overflow answers. We find that: 1) Users performed a much larger than usual revisions on the badge-awarding days compared to normal days; 25% of the users did not make any more revisions once they received their first revision-related badge. 2) Performing more revisions than usual in a single day increased the likelihood of such revisions being rolled back (e.g., due to undesired or incorrect revisions). 3) Users were more likely to perform text and small revisions if they performed many revisions in a single day. Our findings are concurred by the Stack Overflow community, and they highlight the need for changes to the current badge system in order to provide a better balance between the quality and quantity of revisions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Improvements on Cross-Validation: The 632+ Bootstrap Method
Efron, B., & Tibshirani, R. (1997) · Journal of the American Statistical Association · doi:10.1080/01621459.1997.10474007 · Cited by 1,636
Qualitative methods in empirical studies of software engineering
Seaman, C. B. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799955 · Cited by 1,493
Dominance statistics: Ordinal analyses to answer ordinal questions.
Cliff, N. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.114.3.494 · Cited by 1,485
Finding high-quality content in social media
Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008) · Proceedings of the international conference on Web search and web data mining - WSDM '08 · doi:10.1145/1341531.1341557 · Cited by 1,230
Social coding in GitHub: transparency and collaboration in an open software repository
Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145396 · Cited by 999
Improvements on Cross-Validation: The .632+ Bootstrap Method
Efron, B., & Tibshirani, R. J. (1997) · Journal of the American Statistical Association · doi:10.2307/2965703 · Cited by 723
Harnessing the wisdom of crowds in wikipedia: quality through coordination
Kittur, A., & Kraut, R. E. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460572 · Cited by 680
Expectations, outcomes, and challenges of modern code review
Bacchelli, A., & Bird, C. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606617 · Cited by 650
Predictors of answer quality in online Q&A sites
Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357191 · Cited by 412
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
Lowry, P. B., Curtis, A., & Lowry, M. R. (2004) · Journal of Business Communication · doi:10.1177/0021943603259363 · Cited by 365
Steering user behavior with badges
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488398 · Cited by 357
Badges in Social Media: A Social Psychological Perspective
Antin, J., & Churchill, E. F. (2011) · Cited by 297
Measuring article quality in wikipedia: models and evaluation
Hu, M., Lim, E.-P., Sun, A., Lauw, H. W., & Vuong, B.-Q. (2007) · Proceedings of the sixteenth ACM conference on Conference on information and knowledge management · doi:10.1145/1321440.1321476 · Cited by 242
Understanding Statistical Process Control
Ziegel, E. R. (1993) · Technometrics · doi:10.1080/00401706.1993.10485025 · Cited by 240
Answering questions about unanswered questions of Stack Overflow
Asaduzzaman, M., Mashiyat, A. S., Roy, C. K., & Schneider, K. A. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624015 · Cited by 220
Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity
Gwet, K. L. (2002) · Cited by 195
Improving Low Quality Stack Overflow Post Detection
Ponzanelli, L., Mocci, A., Bacchelli, A., Lanza, M., & Fullerton, D. A. (2014) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2014.90 · Cited by 150
How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow
Calefato, F., Lanubile, F., & Novielli, N. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.10.009 · Cited by 149
Collaborative Writing Support Tools on the Cloud
Calvo, R. A., O'Rourke, S. T., Jones, J., Yacef, K., & Reimann, P. (2011) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2010.43 · Cited by 146
Encouraging user behaviour with achievements: An empirical study
Grant, S., & Betts, B. (2013) · 2013 10th Working Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2013.6624007 · Cited by 116
An empirical study of dormant bugs
Chen, T.-H. P., Nagappan, M., Shihab, E., & Hassan, A. E. (2014) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2597073.2597108 · Cited by 84
Self-citationCan Gamification Motivate Voluntary Contributions?: The Case of StackOverflow Q&A Community
Cavusoglu, H., Li, Z., & Huang, K.-W. (2015) · Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2685553.2698999 · Cited by 81
Detecting high-quality posts in community question answering sites
Yao, Y., Tong, H., Xie, T., Akoglu, L., Xu, F., & Lü, J. (2015) · Information Sciences · doi:10.1016/j.ins.2014.12.038 · Cited by 81
Digital badges – rewards for learning?
Shields, R., & Chugh, R. (2016) · Education and Information Technologies · doi:10.1007/s10639-016-9521-x · Cited by 80
Wang, S., Chen, T.-H. P., & Hassan, A. E. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9558-5 · Cited by 79
Self-citationSeaman, C. B., Shull, F. J., Regardie, M., Elbert, D., Feldmann, R. L., Guo, Y., & Godfrey, S. (2008) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1414004.1414030 · Cited by 77
Understanding and Classifying the Quality of Technical Forum Questions
Ponzanelli, L., Mocci, A., Bacchelli, A., & Lanza, M. (2014) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2014.27 · Cited by 77
Why pay?: exploring how financial incentives are used for question & answer
Hsieh, G., Kraut, R. E., & Hudson, S. E. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753373 · Cited by 73
Open Source-Style Collaborative Development Practices in Commercial Projects Using GitHub
Kalliamvakou, E., Damian, D., Blincoe, K., Singer, L., & German, D. M. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.74 · Cited by 72
Quality Questions Need Quality Code: Classifying Code Fragments on Stack Overflow
Duijn, M., Kucera, A., & Bacchelli, A. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.51 · Cited by 65
Li, G., Zhu, H., Lu, T., Ding, X., & Gu, N. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675155 · Cited by 57
Ford, D., Lustig, K., Banks, J., & Parnin, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174182 · Cited by 55
Escalation Behavior as a Specific Case of Goal-Directed Activity: A Persistence Paradigm
Fox, S., & Hoffman, M. (2002) · Basic and Applied Social Psychology · doi:10.1207/s15324834basp2404_3 · Cited by 54
Answer Quality Characteristics and Prediction on an Academic Q&A Site: A Case Study on ResearchGate
Li, L., He, D., Jeng, W., Goodwin, S., & Zhang, C. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2740908.2742129 · Cited by 53
Mining Successful Answers in Stack Overflow
Calefato, F., Lanubile, F., Marasciulo, M. C., & Novielli, N. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.56 · Cited by 45
An Insight into the Unresolved Questions at Stack Overflow
Rahman, M. M., & Roy, C. K. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.55 · Cited by 35
Want a Good Answer? Ask a Good Question First!Preprint
Yao, Y., Tong, H., Xie, T., Akoglu, L., Xu, F., & Lü, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1311.6876 · Cited by 34
Collaborative editing for improved usefulness and usability of transcript-enhanced webcasts
Munteanu, C., Baecker, R., & Penn, G. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357117 · Cited by 32
Automated Verification of Load Tests Using Control Charts
Nguyen, T., Adams, B., Jiang, Z. M., Hassan, A. E., Nasser, M., & Flora, P. (2011) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2011.59 · Cited by 29
Self-citationAn empirical study on the practice of maintaining object-relational mapping code in Java systems
Chen, T.-H. P., Shang, W., Yang, J., Hassan, A. E., Godfrey, M. W., Nasser, M., & Flora, P. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2901758 · Cited by 29
Self-citationWe don't need no stinkin' badges: examining the social role of badges in the Huffington Post
Jones, J., & Altadonna, N. (2012) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2145204.2145244 · Cited by 18
Understanding triggers for clarification requests in community-based software help forums
Hudson, N., Chilana, P. K., Guo, X., Day, J. A., & Liu, E. (2015) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357216 · Cited by 12
Analyzing Payment Based Question and Answering Service.Preprint
Jan, S. T. K., Chun, W., Qing, Z., & Wang, G. (2017) · arXiv (Cornell University) · Cited by 3
Research works printed but not linked: 2
- boslaugh, 2008, Statistics in a Nutshell a Desktop Quick Reference
- 10.1037/0033-2909.114.3.494 · doi:10.1037/0033-2909.114.3.494 · Paper page
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
“Researchers have also studied challenges developers face [14, 23, 32, 58, 72, 74], including difficulties in effective search-query writing and information foraging, and built tools to support developers overcome such challenges [3, 42, 52].”
BackgroundAn Empirical Study of Obsolete Answers on Stack Overflow
Zhang, H., Wang, S., Chen, T.-H. P., Zou, Y., & Hassan, A. E. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2906315 · Cited by 109
Self-citationGitHub Discussions: An exploratory study of early adoption
Hata, H., Novielli, N., Baltes, S., Kula, R. G., & Treude, C. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-10058-6 · Cited by 73
A Study of C/C++ Code Weaknesses on Stack Overflow
Zhang, H., Wang, S., Li, H., Chen, T.-H. P., & Hassan, A. E. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3058985 · Cited by 69
Self-citationReading Answers on Stack Overflow: Not Enough!
Zhang, H., Wang, S., Chen, T.-H. P., & Hassan, A. E. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2954319 · Cited by 66
Self-citation
Metadata
- Authors
- Wang, S., Chen, T.-H., & Hassan, A. E.
- Affiliations
- Queen's University (CA)
- Concordia University (CA)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2020-09-01
- Volume
- 46
- Issue
- 9
- Pages
- 1024-1038
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2897316866
- Cited by
- 50
- References
- 43
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3 · 2023: 5 · 2022: 7 · 2021: 17 · 2020: 7
- FWCI
- 11.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Communication › Wikis in Education and Collaboration
- Keywords
- Stack Overflow, gamification, incentive mechanisms, user behavior, knowledge sharing communities
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.