Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems
Open accessAbstract
On Wikipedia, sophisticated algorithmic tools are used to assess the quality of edits and take corrective actions. However, algorithms can fail to solve the problems they were designed for if they conflict with the values of communities who use them. In this study, we take a Value-Sensitive Algorithm Design approach to understanding a community-created and -maintained machine learning-based algorithm called the Objective Revision Evaluation System (ORES)---a quality prediction system used in numerous Wikipedia applications and contexts. Five major values converged across stakeholder groups that ORES (and its dependent applications) should: (1) reduce the effort of community maintenance, (2) maintain human judgement as the final authority, (3) support differing peoples' differing workflows, (4) encourage positive engagement with diverse editor groups, and (5) establish trustworthiness of people and algorithms within the community. We reveal tensions between these values and discuss implications for future research to improve algorithms like ORES.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large collaborative communities like Wikipedia require AI systems that balance multiple aspects, e.g., reducing human effort, improving trust and positive engagement.
“research shows that large collaborative communities like Wikipedia require AI systems that balance multiple aspects, e.g. , reducing human effort, improving trust and positive engagement [68].”
From Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (2021) · cites this paper as [68]
There exist diverse rules and preferences of austerity and value in different communities.
“there exist diverse rules and preferences of austerity and value in diferent communities [20,31,74,77]”
From Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation (2022) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Interviewing as qualitative research a guide for researchers in education and the social sciences
Seidman, I. (2006) · Cited by 9,900 · see citing works
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · see citing works
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · see citing works
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
Turkopticon: interrupting worker invisibility in amazon mechanical turk
Irani, L. C., & Silberman, M. S. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470742 · Cited by 798 · see citing works
Teacher-Student Matching and the Assessment of Teacher Effectiveness
Clotfelter, C. T., Ladd, H. F., & Vigdor, J. L. (2006) · Journal of Human Resources · doi:10.3368/jhr.xli.4.778 · Cited by 739 · see citing works
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · see citing works
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687 · see citing works
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534 · see citing works
On Fairness and CalibrationPreprint
Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J. M., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02012 · Cited by 509 · see citing works
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490 · cited 1× · see citing works
“However, Veale et al pointed out that these approaches and tools are often built in isolation both from specific users and user contexts [56].”
BackgroundFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375 · see citing works
Creating, destroying, and restoring value in wikipedia
Priedhorsky, R., Chen, J., Lam, S. K., Panciera, K. A., Terveen, L., & Riedl, J. (2007) · International ACM SIGGROUP Conference on Supporting Group Work (GROUP) · doi:10.1145/1316624.1316663 · Cited by 373 · see citing works
Self-citationThe Frontiers of Fairness in Machine LearningPreprint
Chouldechova, A., & Roth, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08810 · Cited by 372 · see citing works
Next steps for value sensitive design
Borning, A., & Muller, M. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208560 · Cited by 365 · see citing works
A Reductions Approach to Fair ClassificationPreprint
Agarwal, A., Beygelzimer, A., Dudı́k, M., Langford, J. C., & Wallach, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.02453 · Cited by 364 · see citing works
Moderator engagement and community development in the age of algorithms
Seering, J., Wang, T., Yoon, J., & Kaufman, G. (2019) · New Media & Society · doi:10.1177/1461444818821316 · Cited by 351 · see citing works
Decentralization in wikipedia governance.
Forte, A., Larco, V., & Bruckman, A. (2009) · Journal of Management Information Systems · doi:10.2753/mis0742-1222260103 · Cited by 300 · see citing works
DeVito, M. A. (2016) · Digital Journalism · doi:10.1080/21670811.2016.1178592 · Cited by 295 · see citing works
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292 · see citing works
Self-citationThe Rise and Decline of an Open Collaboration System
Halfaker, A. L., Geiger, R., Morgan, J. T., & Riedl, J. (2013) · American Behavioral Scientist · doi:10.1177/0002764212469365 · Cited by 267 · see citing works
Self-citationHargittai, E., & Shaw, A. (2015) · Information, Communication & Society · doi:10.1080/1369118x.2014.957711 · Cited by 257 · see citing works
SuggestBot: using intelligent task routing to help people find work in wikipedia
Cosley, D., Frankowski, D., Terveen, L., & Riedl, J. (2007) · Proceedings of the 12th international conference on Intelligent user interfaces · doi:10.1145/1216295.1216309 · Cited by 245 · see citing works
Self-citationValue-Sensitive Algorithm Design: Method, Case Study, and Lessons
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274463 · Cited by 220 · see citing works
Self-citationHalfaker, A., Kittur, A., & Riedl, J. (2011) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/2038558.2038585 · Cited by 206 · see citing works
Self-citationAn Empirical Study of Rich Subgroup Fairness for Machine Learning
Kearns, M. J., Neel, S., Roth, A., & Wu, Z. S. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287592 · Cited by 196 · see citing works
The cost of fairness in binary classification
Menon, A. K., & Williamson, R. C. (2018) · Cited by 157 · see citing works
Crossmod: A Cross-Community Learning-based System to Assist Reddit Moderators
Chandrasekharan, E., Gandhi, C., Mustelier, M. W., & Gilbert, E. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359276 · Cited by 151 · cited 1× · see citing works
“As a first step in this direction, we have identified studies that examined the role of “moderation” in various contexts, including the Turkopticon community of Amazon Mechanical Turk workers [39] and an assortment of communities organized as Reddit subreddits [7, 42], or Twitch channels and Facebook groups [52].”
BackgroundEarly engagement and new technologies: Opening up the laboratory
Doorn, N., Gorman, M. E., van de Poel, I., & Schuurbiers, D. (2013) · Philosophy of engineering and technology · doi:10.1007/978-94-007-7844-3 · Cited by 136 · see citing works
Bots, bespoke, code and the materiality of software platforms
Geiger, R. S. (2014) · Information Communication & Society · doi:10.1080/1369118x.2013.873069 · Cited by 130 · see citing works
A five-year study of on-campus Internet use by undergraduate biomedical students
Judd, T. S., & Kennedy, G. E. (2010) · Computers & Education · doi:10.1016/j.compedu.2010.06.022 · Cited by 108 · see citing works
Halfaker, A., Kittur, A., Kraut, R. E., & Riedl, J. (2009) · International Symposium on Wikis and Open Collaboration (WikiSym) · doi:10.1145/1641309.1641332 · Cited by 95 · see citing works
Self-citationHuman intervention in automated decision-making: Toward the construction of contestable systems
Almada, M. (2019) · Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law · doi:10.1145/3322640.3326699 · Cited by 95 · see citing works
Geiger, R. S. (2017) · Big Data & Society · doi:10.1177/2053951717730735 · Cited by 92 · see citing works
Surviving an "Eternal September"
Kiene, C., Monroy‐Hernández, A., & Hill, B. M. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858356 · Cited by 90 · see citing works
A Human-Centered Approach to Algorithmic Services
Lee, M. K., Kim, J. T., & Lizarondo, L. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025884 · Cited by 88 · see citing works
When the levee breaks: without bots, what happens to Wikipedia's quality control processes?
Geiger, R. S., & Halfaker, A. (2013) · Proceedings of the 9th International Symposium on Open Collaboration · doi:10.1145/2491055.2491061 · Cited by 67 · see citing works
Self-citationBots and Cyborgs: Wikipedia's Immune System
Halfaker, A., & Riedl, J. (2012) · Computer · doi:10.1109/mc.2012.82 · Cited by 62 · see citing works
Self-citationWork-to-rule: the emergence of algorithmic governance in Wikipedia
Müller-Birn, C., Dobusch, L., & Herbsleb, J. D. (2013) · Proceedings of the 6th International Conference on Communities and Technologies · doi:10.1145/2482991.2482999 · Cited by 61 · see citing works
What's the Folk Theory? Reasoning About Cyber-Social SystemsPreprint
French, M., & Hancock, J. T. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2910571 · Cited by 58 · see citing works
Quality Assessment of Wikipedia Articles without Feature Engineering
Dang, Q.-V., & Ignat, C. (2016) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/2910896.2910917 · Cited by 57 · see citing works
Examining Wikipedia With a Broader Lens
Vincent, N., Johnson, I. L., & Hecht, B. J. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174140 · Cited by 45 · see citing works
Interpolating Quality Dynamics in Wikipedia and Demonstrating the Keilana Effect
Halfaker, A. (2017) · International Symposium on Open Collaboration · doi:10.1145/3125433.3125475 · Cited by 43 · see citing works
Self-citationOperationalizing Conflict and Cooperation between Automated Software Agents in Wikipedia
Geiger, R. S., & Halfaker, A. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134684 · Cited by 35 · see citing works
Self-citationDeVito, M. A., Hancock, J. T., French, M., Birnholtz, J. P., Antin, J., Karahalios, K. G., Tong, S. T., & Shklovski, I. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3186320 · Cited by 33 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- R. Stuart Geiger. 2011. The lives of bots. In Critical point of view: a Wikipedia reader, Geert Lovink and Nathaniel Tkacz (Eds.). Number 7 in INC Reader. Inst. of Network Cultures, Amsterdam, 78-93. http: //stuartgeiger.com/lives-of-bots-wikipedia-cpov.pdf OCLC: 762034235.
- Aaron Halfaker, R Stuart Geiger, Jonathan T Morgan, Amir Sarabadani, and Adam Wight. 2018. ORES: Facilitating re-mediation of Wikipedia's socio-technical problems. Computer-Supported Cooperative Work and Social Computing (CSCW) (2018).
- Tom Simonite. 2013. The decline of Wikipedia. Technology Review 116, 6 (2013), 50-56.
- Lauren Weber and RE Silverman. 2012. Your resume vs. oblivion. The Wall Street Journal 24 (2012).
- Wikipedia. 2019. Help:New filters for edit review/Quality and Intent Filters -MediaWiki. (2019). https://www.mediawiki.org/wiki/Help:New_filters_for_ edit_review/Quality_and_Intent_Filters
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · see citing works
“In fact, research shows that large collaborative communities such as Wikipedia require AI systems that balance multiple aspects such as reducing human effort, improving trust and positive engagement [58].”
BackgroundThe Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
Delgado, F., Yang, S., Madaio, M., & Yang, Q. (2023) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3617694.3623261 · Cited by 256 · see citing works
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · see citing works
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · see citing works
Self-citationPublic engagement and AI: A values analysis of national strategies
Wilson, C. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2021.101652 · Cited by 140 · see citing works
Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Gordon, M., Lam, M. S., Park, J.-S., Patel, K., Hancock, J. T., Hashimoto, T., & Bernstein, M. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502004 · Cited by 143 · see citing works
Stapleton, L., Lee, M. H., Qing, D., Wright, M., Chouldechova, A., Holstein, K., Wu, Z. S., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533177 · Cited by 86 · see citing works
Self-citationORES: Lowering Barriers with Participatory Machine Learning in Wikipedia
Halfaker, A., & Geiger, R. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415219 · Cited by 82 · see citing works
Self-citationLam, M. S., Gordon, M., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 78 · see citing works
Park, H., Ahn, D., Hosanagar, K., & Lee, J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517672 · Cited by 79 · see citing works
Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579601 · Cited by 77 · see citing works
Robertson, S., Nguyen, T., & Salehi, N. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445748 · Cited by 67 · see citing works
Power and Public Participation in AI
Corbett, E., Denton, E., & Erete, S. L. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623228 · Cited by 57 · see citing works
Varanasi, R. A., & Goyal, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580903 · Cited by 52 · see citing works
van Berkel, N., Papachristos, E., Giachanou, A., Hosio, S., & Skov, M. B. (2020) · Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society · doi:10.1145/3419249.3420106 · Cited by 51 · see citing works
The Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design
Shen, H., Wang, L., Deng, W. H., Brusse, C., Velgersdijk, R., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533110 · Cited by 45 · see citing works
Self-citationvan Berkel, N., Sarsenbayeva, Z., & Gonçalves, J. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102954 · Cited by 41 · see citing works
Smith, C. E., Lane, W., Miller, H. J., Kluver, D., Terveen, L., & Yarosh, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479561 · Cited by 37 · see citing works
Self-citationWright, A. P., Shaikh, O., Park, H., Epperson, W., Ahmed, M., Pinel, S., Chau, D. H. (., & Yang, D. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449280 · Cited by 35 · see citing works
Epistemic values in feature importance methods
Hancox-Li, L., & Kumar, I. E. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445943 · Cited by 36 · see citing works
Public Health Calls for/with AI: An Ethnographic Perspective
Ismail, A., Thakkar, D., Madhiwalla, N., & Kumar, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610203 · Cited by 36 · see citing works
From Preference Elicitation to Participatory ML: A Critical Survey & Guidelines for Future Research
Feffer, M., Skirpan, M. W., Lipton, Z. C., & Heidari, H. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604661 · Cited by 27 · see citing works
Asthana, S., Hilleli, S., He, P., & Halfaker, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711074 · Cited by 38 · see citing works
Self-citationKawakami, A., Coston, A., Heidari, H., Holstein, K., & Zhu, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686989 · Cited by 24 · see citing works
Self-citationStakeholder Involvement for Responsible AI Development: A Process Framework
Kallina, E., & Singh, J. (2024) · Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3689904.3694698 · Cited by 23 · see citing works
Ren, Y., Zhang, H., & Kraut, R. E. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617369 · Cited by 23 · see citing works
Yoo, D. W., Woo, H., Pendse, S. R., Lu, N. Y., Birnbaum, M. L., Abowd, G. D., & Choudhury, M. D. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637372 · Cited by 22 · see citing works
ORES: Lowering Barriers with Participatory Machine Learning in WikipediaPreprint
Halfaker, A., & Geiger, R. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05189 · Cited by 22 · see citing works
Self-citationWikibench: Community-Driven Data Curation for AI Evaluation on Wikipedia
Kuo, T.-S., Halfaker, A., Cheng, Z., Kim, J., Wu, M.-H., Wu, T., Holstein, K., & Zhu, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642278 · Cited by 20 · see citing works
Self-citationThe Medical Authority of AI: A Study of AI-enabled Consumer-Facing Health Technology
You, Y., Kou, Y., Ding, X., & Gui, X. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445657 · Cited by 18 · see citing works
Can bots help create knowledge? The effects of bot intervention in open collaboration
Kang, S., Liu, X., Kim, Y., & Yoon, V. (2021) · Decision Support Systems · doi:10.1016/j.dss.2021.113601 · Cited by 18 · see citing works
Language-agnostic Topic Classification for Wikipedia
Johnson, I. L., Gerlach, M., & Sáez-Trumper, D. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442442.3452347 · Cited by 18 · see citing works
Challenges in Value-Sensitive AI Design: Insights from AI Practitioner Interviews
Sadek, M., & Mougenot, C. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2439021 · Cited by 17 · see citing works
Smith, C. E., Miller, H. J., & Levonian, Z. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610035 · Cited by 17 · see citing works
Self-citationPrinster, G. H., Smith, C. E., Tan, C., & Keegan, B. C. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3637310 · Cited by 23 · see citing works
Self-citationPark, J.-S., Karahalios, K. G., Salehi, N., & Eslami, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512957 · Cited by 16 · see citing works
Disseminating Research News in HCI
Smith, C. E., Nevarez, E., & Zhu, H. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376744 · Cited by 20 · see citing works
Self-citation“People are Way too Obsessed with Rank”: Trust System in Social Virtual Reality
Chen, Q., Cai, J., & Jacucci, G. (2024) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-024-09498-7 · Cited by 16 · see citing works
Kuo, T.-S., Chen, Q. Z., Zhang, A. X., Hsieh, J., Zhu, H., & Holstein, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713865 · Cited by 12 · see citing works
Self-citationThe Impact of Governance Bots on Sense of Virtual Community
Smith, C. E., Alam, I., Tan, C., Keegan, B., & Blanchard, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555563 · Cited by 12 · see citing works
Self-citationCura: Curation at Social Media Scale
He, W., Gordon, M., Popowski, L., & Bernstein, M. S. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610186 · Cited by 22 · see citing works
Wang, L., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533147 · Cited by 13 · see citing works
Self-citationConvEx: A Visual Conversation Exploration System for Discord Moderators
Choi, F., Bajpai, T., Pratipati, S., & Chandrasekharan, E. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610053 · Cited by 27 · see citing works
Observe, Ask, Intervene: Designing AI Agents for More Inclusive Meetings
Houtti, M., Zhou, M., Terveen, L., & Chancellor, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713838 · Cited by 12 · see citing works
Self-citationSecondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency
Ajmani, L., Abdelkadir, N. A., & Chancellor, S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732071 · Cited by 11 · see citing works
Sharma, T., Potter, Y., Pongmala, K., Wang, H., Miller, A., Song, D., & Wang, Y. (2024) · ACM Transactions on the Web · doi:10.1145/3796547 · Cited by 19 · see citing works
"Nip it in the Bud": Moderation Strategies in Open Source Software Projects and the Role of Bots
Hsieh, J., Kim, J., Dabbish, L., & Zhu, H. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610092 · Cited by 11 · see citing works
Self-citationTrokhymovych, M., & Sáez-Trumper, D. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481961 · Cited by 11 · see citing works
Value Tensions in OpenStreetMap: Openness, Membership, and Policy in Online Communities
Chauhan, A., Sarkar, D., Agrawaal, T. S., & Soden, R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686919 · Cited by 10 · see citing works
An endangered species: how LLMs threaten Wikipedia’s sustainability
Vetter, M. A., Jiang, J., & McDowell, Z. J. (2025) · AI & Society · doi:10.1007/s00146-025-02199-9 · Cited by 9 · see citing works
Representation of Music Creators on Wikipedia, Differences in Gender and Genre
Wang, A., Pappu, A., & Cramer, H. S. M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v15i1.18101 · Cited by 9 · see citing works
Wikipedia ORES Explorer: Visualizing Trade-offs For Designing Applications With Machine Learning API
Ye, Z., Yuan, X., Gaur, S., Halfaker, A., Forlizzi, J., & Zhu, H. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462099 · Cited by 9 · see citing works
Self-citationZhang, C. C., Houtti, M., Smith, C. E., Kong, R., & Terveen, L. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512982 · Cited by 9 · see citing works
Self-citationTowards Fair and Equitable Incentives to Motivate Paid and Unpaid Crowd Contributions
Wallace, S., Massachi, T., Su, J., Miller, D., & Huang, J. J. S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714195 · Cited by 8 · see citing works
Deng, W. H., Barocas, S., & Vaughan, J. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711076 · Cited by 12 · see citing works
Wallace, S., Papoutsaki, A., Tan, N. H., Guo, H., & Huang, J. J. S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479592 · Cited by 8 · see citing works
Chancellor, S., Feuston, J. L., & Chang, J. (2023) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3610181 · Cited by 12 · see citing works
Confrontation and Cultivation: Understanding Perspectives on Robot Responses to Norm Violations
Mott, T., & Williams, T. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309577 · Cited by 7 · see citing works
Disseminating Research News in HCI: Perceived Hazards, How-To's, and Opportunities for InnovationPreprint
Smith, C. E., Nevarez, E., & Zhu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04883 · Cited by 7 · see citing works
Self-citationTemplates and Trust-o-meters: Towards a widely deployable indicator of trust in Wikipedia
Kuznetsov, A. V., Novotny, M., Klein, J., Sáez-Trumper, D., & Kittur, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517523 · Cited by 6 · see citing works
Collaborating with Bots and Automation on OpenStreetMap
van Berkel, N., & Pohl, H. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3665326 · Cited by 5 · see citing works
Robertson, S., Nguyen, T., Hu, C., Albiston, C. R., Nikzad, A., & Salehi, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580996 · Cited by 5 · see citing works
Exploring the impact of ChatGPT on Wikipedia engagement
Reeves, N., Yin, W., & Simperl, E. P. B. (2025) · Collective Intelligence · doi:10.1177/26339137251372599 · Cited by 10 · see citing works
Hellman, J., Epstein, I., Cheng, J., & Guo, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757443 · Cited by 3 · see citing works
Investigating mental wellbeing self-care in higher education using BERTopic modeling
Ali, M., van Berkel, N., Tag, B., Paananen, V., Oppenlaender, J., Yatani, K., & Hosio, S. J. (2025) · Discover Mental Health · doi:10.1007/s44192-025-00323-1 · Cited by 6 · see citing works
Oguine, O. C., Garcia, A. A., Muller, M. J., & Badillo-Urquiola, K. A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791346 · Cited by 4 · see citing works
InfoSync: Information Synchronization across Multilingual Semi-structured Tables
Khincha, S., Jain, C., Gupta, V., Kataria, T., & Zhang, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.159 · Cited by 2 · see citing works
Do Know Harm: Considering the Ethics of Online Community Research
Chancellor, S., Konstan, J. A., Terveen, L., & Yarosh, S. (2024) · Communications of the ACM · doi:10.1145/3635303 · Cited by 2 · see citing works
Self-citationAnchor Prediction: A Topic Modeling Approach
Dupuy, J., Guille, A., & Jacques, J. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524927 · Cited by 2 · see citing works
How Does Predictive Information Affect Human Ethical Preferences?
Narayanan, S., Yu, G., Tang, W., Ho, C.-J., & Yin, M. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534165 · Cited by 2 · see citing works
WikiCheck: An end-to-end open source Automatic Fact-Checking API based on WikipediaPreprint
Trokhymovych, M., & Sáez-Trumper, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00835 · Cited by 2 · see citing works
Fair Multilingual Vandalism Detection System for Wikipedia
Trokhymovych, M., Aslam, M., Chou, A.-J., Baeza-Yates, R. A., & Sáez-Trumper, D. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599823 · Cited by 3 · see citing works
Machines in the Margins: A Systematic Review of Automated Content Generation for Wikipedia
Reeves, N., & Simperl, E. P. B. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757626
Yu, Y., & McDonald, D. W. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816912
Sanctioned by Others: Remediating Knowledge Exclusion on Wikipedia Through Design
Ajmani, L., Zhou, M., Andic, T., Berson, A., Johnson, I. L., Hill, B. M., Maung, R., Narayan, S., Severson, S., Terveen, L., et al. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816994
Self-citationUllstein, C., Hohendanner, M., Großklags, J., Babalola, O., Onyekwelu, B. A., Katirai, A., Ozaki, A., Kuribayashi, J., Ema, A., Sharma, N., et al. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816960
- Authors
- Smith, C. E., Yu, B., Srivastava, A., Halfaker, A., Terveen, L., & Zhu, H.
- Affiliations
- University of Minnesota (US)
- Wikimedia Foundation (US)
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-14
- OpenAlex
- W2998862821
- Cited by
- 77
- References
- 51
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 10 · 2023: 16 · 2022: 15 · 2021: 15 · 2020: 4 · 2019: 1
- FWCI
- 12.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (IIS-2001851,IIS-2000782,IIS-1939606)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Communication › Wikis in Education and Collaboration
- Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
- Keywords
- Wikipedia, ore rock, stakeholder value, community values, algorithmic trust, community participation, algorithmic bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
