A Checklist to Combat Cognitive Biases in Crowdsourcing
Open accessAbstract
Recent research has demonstrated that cognitive biases such as the confirmation bias or the anchoring effect can negatively affect the quality of crowdsourced data. In practice, however, such biases go unnoticed unless specifically assessed or controlled for. Task requesters need to ensure that task workflow and design choices do not trigger workers’ cognitive biases. Moreover, to facilitate the reuse of crowdsourced data collections, practitioners can benefit from understanding whether and which cognitive biases may be associated with the data. To this end, we propose a 12-item checklist adapted from business psychology to combat cognitive biases in crowdsourcing. We demonstrate the practical application of this checklist in a case study on viewpoint annotations for search results. Through a retrospective analysis of relevant crowdsourcing research that has been published at HCOMP in 2018, 2019, and 2020, we show that cognitive biases may often affect crowd workers but are typically not considered as potential sources of poor data quality. The checklist we propose is a practical tool that requesters can use to improve their task designs and appropriately describe potential limitations of collected data. It contributes to a body of efforts towards making human-labeled data more reliable and reusable.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Humans are prone to cognitive biases.
“According to the checklist of cognitive biases provided by Draws et al. [33], it is important to acknowledge that humans are prone to cognitive biases.”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as Draws et al. [33]
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · cited 5× · see citing works
“Cognitive biases are human tendencies towards irrational decision-making or deviation from norms (Tversky and Kahneman 1974).”
InfluentialJudgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
Jeffreys, H., Moran, P. A. P., & Chatfield, C. (1922) · Nature · doi:10.1038/109132a0 · Cited by 7,837 · see citing works
Kruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731 · see citing works
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267 · see citing works
(2023) · doi:10.1126/science · Cited by 6,063 · see citing works
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200 · see citing works
A tutorial on regularized partial correlation networks
Epskamp, S., & Fried, E. I. (2018) · Psychological Methods · doi:10.1037/met0000167 · Cited by 2,918 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
The preregistration revolution
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708274114 · Cited by 1,998 · see citing works
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887 · see citing works
Bayesian Cognitive Modeling: A Practical Course
Lee, M. D., & Wagenmakers, E.-J. (2014) · doi:10.1017/cbo9781139087759 · Cited by 1,795 · see citing works
Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications
Wagenmakers, E., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., Gronau, Q. F., Šmíra, M., Epskamp, S., et al. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1343-3 · Cited by 1,652 · see citing works
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206 · see citing works
The Checklist Manifesto: How to Get Things Right
Gawande, A. A. (2011) · Journal of Nursing Regulation · doi:10.1016/s2155-8256(15)30310-0 · Cited by 1,155 · see citing works
Four assumptions of multiple regression that researchers should always test
Osbourne, J. W., & Waters, E. (2020) · Scholarworks (University of Massachusetts Amherst) · doi:10.7275/r222-hv23 · Cited by 1,042 · see citing works
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · see citing works
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F. J., & Wang, J. (2010) · doi:10.1145/1837885.1837906 · Cited by 991 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 2× · see citing works
“In the NLP field, Bender and Friedman (2018) proposed data statements , a characterization for data sets that provides relevant details regarding the population involved in creating a given data set, how the data set is used in experimental work, and how potential biases in the data set might affect…”
MethodFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Hilbert, M. (2012) · Psychological Bulletin · doi:10.1037/a0025940 · Cited by 391 · cited 2× · see citing works
“However, a plethora of different cognitive biases have been identified to date (Hilbert 2012), and for many cognitive biases, it is still unclear whether or how they may affect crowd workers.”
BackgroundShepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · see citing works
Before you make that big decision...
Kahneman, D. J., Lovallo, D. P., & Sibony, O. (2011) · PubMed · Cited by 332 · see citing works
Geva, M., Goldberg, Y., & Berant, J. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1107 · Cited by 284 · see citing works
Predictive biases in natural language processing models: A conceptual framework and overview
Shah, D. S., Schwartz, H. A., & Hovy, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.468 · Cited by 241 · see citing works
ULLMAN, J. B., & Bentler, P. M. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9781848608184.n19 · Cited by 235 · see citing works
The Preregistration RevolutionPreprint
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017) · doi:10.31219/osf.io/2dxu5 · Cited by 220 · see citing works
(2019) · doi:10.1145/3351095.3372862 · Cited by 167 · see citing works
Incentivizing High Quality Crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741102 · Cited by 146 · see citing works
Cognitive Biases in Crowdsourcing
Eickhoff, C. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159654 · Cited by 144 · see citing works
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129 · see citing works
Self-citationThe checklist manifesto: how to get things right
Chew, T. S. (2011) · Clinical Medicine · doi:10.7861/clinmedicine.11-3-296 · Cited by 128 · see citing works
What's the right price? pricing tasks for finishing on time
Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011) · Cited by 126 · see citing works
Social desirability bias and self-reports of motivation
Antin, J., & Shaw, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208699 · Cited by 125 · see citing works
Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing
Mao, A., Kamar, E. S., Chen, Y., Horvitz, E., Schwamb, M. E., Lintott, C. J., & Smith, A. M. (2013) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v1i1.13075 · Cited by 119 · see citing works
A taxonomy of microtasks on the web
Gadiraju, U., Kawase, R., & Dietze, S. (2014) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/2631775.2631819 · Cited by 116 · see citing works
Self-citationIncreasing the Reliability of Crowdsourcing Evaluations Using Online Quality Assessment
Burmania, A., Parthasarathy, S., & Busso, C. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2493525 · Cited by 108 · see citing works
Barbosa, N. M., & Chen, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300773 · Cited by 90 · see citing works
Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms
Difallah, D. E., Demartini, G., & Cudré-Mauroux, P. (2012) · Cited by 90 · see citing works
Clarity is a Worthwhile Quality
Gadiraju, U., Yang, J., & Bozzon, A. (2017) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3078714.3078715 · Cited by 87 · see citing works
Self-citationPogacar, F. A., Ghenai, A., Smucker, M. D., & Clarke, C. L. A. (2017) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3121050.3121074 · Cited by 87 · see citing works
Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
McDonnell, T., Lease, M., Kutlu, M., & Elsayed, T. (2016) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v4i1.13287 · Cited by 84 · see citing works
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions
Aroyo, L. M., Dixon, L., Thain, N., Redfield, O., & Rosen, R. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317083 · Cited by 64 · see citing works
Priming for Better Performance in Microtask Crowdsourcing Environments
Morris, R. R., Dontcheva, M., & Gerber, E. M. (2012) · IEEE Internet Computing · doi:10.1109/mic.2012.68 · Cited by 64 · see citing works
The Impact of Task Abandonment in Crowdsourcing
Han, L., Roitero, K., Gadiraju, U., Sarasua, C., Checco, A., Maddalena, E., & Demartini, G. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2948168 · Cited by 60 · see citing works
Self-citationIncentives to Counter Bias in Human Computation
Faltings, B. V., Jurca, R., Pu, P., & Tran, B. D. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13145 · Cited by 58 · see citing works
Han, L., Roitero, K., Gadiraju, U., Sarasua, C., Checco, A., Maddalena, E., & Demartini, G. (2019) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3289600.3291035 · Cited by 55 · see citing works
Self-citationUsing Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P., & Dietze, S. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3119930 · Cited by 54 · see citing works
Self-citationDaniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02546 · Cited by 52 · see citing works
Rieger, A., Draws, T., Theune, M., & Tintarev, N. (2021) · Proceedings of the 32st ACM Conference on Hypertext and Social Media · doi:10.1145/3465336.3475101 · Cited by 48 · see citing works
Self-citationConfusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk
Wu, M., & Quinn, A. J. (2017) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v5i1.13317 · Cited by 45 · see citing works
What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
Peng, A., Nushi, B., Kıcıman, E., Inkpen, K. M., Suri, S., & Kamar, E. S. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5281 · Cited by 45 · see citing works
Draws, T., Tintarev, N., Gadiraju, U., Bozzon, A., & Timmermans, B. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462851 · Cited by 38 · see citing works
Self-citationNutritional Labels for Data and Models
Stoyanovich, J., & Howe, B. (2019) · IEEE Data(base) Engineering Bulletin · Cited by 36 · see citing works
Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics
Draws, T., Tintarev, N., Gadiraju, U., Bozzon, A., & Timmermans, B. (2020) · SIGKDD Explorations · doi:10.1145/3468507.3468515 · Cited by 36 · see citing works
Self-citationModelling Cognitive Bias in Crowdsourcing Systems
Saab, F., Elhajj, I. H., Kayssi, A. I., & Chehab, A. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.04.004 · Cited by 31 · see citing works
Incentivizing high quality crowdwork
Ho, C.-J., Slivkins, A., Suri, S., & Vaughan, J. W. (2016) · ACM SIGecom Exchanges · doi:10.1145/2904104.2904108 · Cited by 29 · see citing works
CrowdTruth 2.0: Quality Metrics for Crowdsourcing with DisagreementPreprint
Dumitrache, A., Inel, O., Aroyo, L. M., Timmermans, B., & Welty, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06080 · Cited by 24 · see citing works
Self-citationOtterbacher, J. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702151 · Cited by 21 · see citing works
How Do We Talk about Other People? Group (Un)Fairness in Natural Language Image Descriptions
Otterbacher, J., Barlas, P., Kleanthous, S., & Kyriakou, K. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5267 · Cited by 20 · see citing works
Implicit Bias in Crowdsourced Knowledge Graphs
Demartini, G. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317307 · Cited by 19 · see citing works
An Empirical Study on Short- and Long-Term Effects of Self-Correction in Crowdsourced Microtasks
Kobayashi, M., Morita, H., Matsubara, M., Shimizu, N., & Morishima, A. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13324 · Cited by 16 · see citing works
Studying Topical Relevance with Evidence-based Crowdsourcing
Inel, O., Haralabopoulos, G., Li, D., Gysel, C. V., Szlávik, Z., Simperl, E. P. B., Kanoulas, E., & Aroyo, L. M. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3271779 · Cited by 15 · see citing works
Self-citationDetecting cognitive bias in a relevance assessment task using an eye tracker
Harris, C. G. (2019) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3314111.3319824 · Cited by 14 · see citing works
Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition
Mohanty, V., Abdol-Hamid, K., Ebersohl, C., & Luther, K. (2019) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v7i1.5272 · Cited by 12 · see citing works
Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs
Kemmer, R., Yoo, Y., Escobedo, A. R., & Maciejewski, R. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7465 · Cited by 11 · see citing works
Improving Reactions to Rejection in Crowdsourcing Through Self-Reflection
Edixhoven, T., Qiu, S., Kuiper, L., Dikken, O., Smitskamp, G., & Gadiraju, U. (2021) · ACM Web Science Conference (WebSci) · doi:10.1145/3447535.3462482 · Cited by 10 · see citing works
Self-citationRamírez, J., Báez, M., Casati, F., Cernuzzi, L., & Benatallah, B. (2020) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3406865.3418318 · Cited by 8 · see citing works
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices
Ramírez, J., Sayin, B., Baez, M., Casati, F., Cernuzzi, L., Benatallah, B., & Demartini, G. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479531 · Cited by 4 · see citing works
Flagged first, then by citations
5 of the 76 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
He, G., Kuiper, L., & Gadiraju, U. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581025 · Cited by 107 · see citing works
“With the help of Cognitive Biases Checklist introduced [15], we analyzed potential bias in our study.”
BackgroundMethodSelf-citationToward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114 · see citing works
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73 · see citing works
“Previous work has studied how to efficiently use crowdsourcing, with all its limitations, to accomplish a variety of tasks (Draws et al., 2021; Bragg et al., 2013), conducted au-dits on the overall quality of crowdsourced annotations (Chmielewski and Kucker, 2020; Kennedy et al., 2020; Smith et…”
Crowdsourcing Geospatial Data for Earth and Human Observations: A Review
Huang, X., Wang, S., Yang, D., Hu, T., Chen, M., Zhang, M., Zhang, G., Biljecki, F., Lu, T., Zou, L., et al. (2023) · Journal of Remote Sensing · doi:10.34133/remotesensing.0105 · Cited by 71 · see citing works
Cognitive Biases in Fact-Checking and Their Countermeasures: A Review
Soprano, M., Roitero, K., Barbera, D. L., Ceolin, D., Spina, D., Demartini, G., & Mizzaro, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103672 · Cited by 68 · see citing works
Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
He, G., Aishwarya, N., & Gadiraju, U. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712133 · Cited by 53 · see citing works
Self-citationTowards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 42 · see citing works
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
“According to the checklist of cognitive biases provided by Draws et al. [33], it is important to acknowledge that humans are prone to cognitive biases.”
Self-citationDraws, T., Inel, O., Tintarev, N., Baden, C., & Timmermans, B. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505812 · Cited by 29 · see citing works
Self-citationDoes Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED
Huang, Q., Hao, S., Ye, Y., Zhu, S., Feng, Y., & Zhao, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.432 · Cited by 28 · see citing works
Who's in the Crowd Matters: Cognitive Factors and Beliefs Predict Misinformation Assessment Accuracy
Kaufman, R. A., Haupt, M. R., & Dow, S. P. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555611 · Cited by 27 · see citing works
The Effects of Crowd Worker Biases in Fact-Checking Tasks
Draws, T., La Barbera, D., Soprano, M., Roitero, K., Ceolin, D., Checco, A., & Mizzaro, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534629 · Cited by 23 · see citing works
Self-citationAlbers, N., Neerincx, M. A., & Brinkman, W. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277295 · Cited by 22 · see citing works
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 23 · see citing works
Thomas, P., Kazai, G., White, R. W., & Craswell, N. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505815 · Cited by 14 · see citing works
He, G., Balayn, A., Buijsman, S., Yang, J., & Gadiraju, U. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21990 · Cited by 12 · see citing works
Self-citationde Groot, E. C. S., & Gadiraju, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642429 · Cited by 12 · see citing works
Self-citationAkal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology
Hada, R., Husain, S., Gumma, V., Diddee, H., Yadavalli, A., Seth, A., Kulkarni, N., Gadiraju, U., Vashistha, A., Seshadri, V., et al. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659017 · Cited by 12 · see citing works
Self-citationKou, Z., Shang, L., Zhang, Y., Duan, S., & Wang, D. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512256 · Cited by 12 · see citing works
Abbas, T., & Gadiraju, U. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21983 · Cited by 12 · see citing works
Self-citationNORMalize: The First Workshop on Normative Design and Evaluation of Recommender Systems
Vrijenhoek, S., Michiels, L., Kruse, J., Starke, A. D., Tintarev, N., & Guerrero, J. V. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608757 · Cited by 12 · see citing works
Self-citationAlbers, N., Neerincx, M. A., Penfornis, K. M., & Brinkman, W. (2022) · PeerJ · doi:10.7717/peerj.13824 · Cited by 11 · see citing works
The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and Guidelines
Oppenlaender, J., Abbas, T., & Gadiraju, U. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641023 · Cited by 11 · see citing works
Self-citationWhen More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21994 · Cited by 9 · see citing works
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
Inel, O., Draws, T., & Aroyo, L. M. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27547 · Cited by 10 · see citing works
Self-citationCHIME: Causal Human-in-the-Loop Model Explanations
Biswas, S., Corti, L., Buijsman, S., & Yang, J. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21985 · Cited by 9 · see citing works
From Potential to Practice: Intellectual Humility During Search on Debated Topics
Rieger, A., Bredius, F., Theune, M., & Pera, M. S. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638306 · Cited by 9 · see citing works
Self-citationThe Influence of Curiosity Traits and On-Demand Explanations in AI-Assisted Decision-Making
Cau, F. M., & Spano, L. D. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712165 · Cited by 8 · see citing works
Workshop on Understanding and Mitigating Cognitive Biases in Human-AI Collaboration
Boonprakong, N., He, G., Gadiraju, U., van Berkel, N., Wang, D., Chen, S., Liu, J.-Q., Tag, B., Gonçalves, J., & Dingler, T. (2023) · CSCW Companion · doi:10.1145/3584931.3611284 · Cited by 7 · see citing works
Self-citationTrustworthy human computation: a survey
Kashima, H., Oyama, S., Arai, H., & Mori, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10974-1 · Cited by 7 · see citing works
Yurrita, M., Verma, H., Balayn, A., Gadiraju, U., Pont, S. C., & Bozzon, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713145 · Cited by 8 · see citing works
Self-citationZhang, Y., Zong, R., Shang, L., Zeng, H., Yue, Z., Wei, N., & Wang, D. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/701 · Cited by 7 · see citing works
Pradhan, V. K., Schaekermann, M., & Lease, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.828187 · Cited by 7 · see citing works
Assessing the potential of generative agents in crowdsourced fact-checking
Costabile, L., Orlando, G. M., Gatta, V. L., & Moscato, V. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100326 · Cited by 6 · see citing works
Zong, R., Zhang, Y., Stinar, F., Shang, L., Zeng, H., Bosch, N., & Wang, D. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27560 · Cited by 6 · see citing works
News Ninja: Gamified Annotation of Linguistic Bias in Online News
Hinterreiter, S., Spinde, T., Oberdörfer, S., Echizen, I., & Latoschik, M. E. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3677092 · Cited by 6 · see citing works
Busch, P. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100490 · Cited by 11 · see citing works
Moreira, G., Bogucka, E. P., Constantinides, M., & Quercia, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714117 · Cited by 5 · see citing works
Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
Roitero, K., Wright, D., Soprano, M., Augenstein, I., & Mizzaro, S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729960 · Cited by 5 · see citing works
Responsible Opinion Formation on Debated Topics in Web Search
Rieger, A., Draws, T., Mattis, N. M., Maxwell, D., Elsweiler, D., Gadiraju, U., McKay, D., Bozzon, A., & Pera, M. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_32 · Cited by 5 · see citing works
Self-citationHe, Y., Zhang, M., Wu, T., Guo, X., & Vogel, D. (2025) · Electronic Markets · doi:10.1007/s12525-024-00748-6 · Cited by 5 · see citing works
Zhang, Y., Shang, L., Zong, R., Zeng, H., Yue, Z., & Wang, D. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583871 · Cited by 5 · see citing works
Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability
Yurrita, M., Verma, H., Balayn, A., Alfrink, K., Gadiraju, U., & Bozzon, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757415 · Cited by 7 · see citing works
Self-citationMaximizing Neutrality in News Ordering
Advani, R., Papotti, P., & Asudeh, A. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599425 · Cited by 4 · see citing works
BiasX: “Thinking Slow” in Toxic Content Moderation with Explanations of Implied Social Biases
Zhang, Y., Nanduri, S., Jiang, L., Wu, T., & Sap, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.300 · Cited by 11 · see citing works
Salimzadeh, S., Gadiraju, U., Hauff, C., & Arie Van Deursen, A. (2022) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3511095.3531282 · Cited by 4 · see citing works
Self-citationSoprano, M., Roitero, K., Gadiraju, U., Maddalena, E., & Demartini, G. (2024) · ACM Transactions on Social Computing · doi:10.1145/3674884 · Cited by 4 · see citing works
Self-citationSignUpCrowd: Using Sign-Language as an Input Modality for Microtask Crowdsourcing
Singh, A., Wehkamp, S., & Gadiraju, U. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21998 · Cited by 3 · see citing works
Self-citationCascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models
Malaviya, C., Bhatia, S., & Yatskar, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.438 · Cited by 3 · see citing works
Radio Galaxy Zoo: Tagging radio subjects using text
Chen, D., Kerai, V., Alger, M. J., Wong, O. I., & Ong, C. S. (2023) · Publications of the Astronomical Society of Australia · doi:10.1017/pasa.2023.50 · Cited by 3 · see citing works
Barbera, D. L., Roitero, K., & Mizzaro, S. (2023) · Intelligenza Artificiale · doi:10.3233/ia-230011 · Cited by 3 · see citing works
NORMalize: A Tutorial on the Normative Design and Evaluation of Information Access Systems
Kruse, J., Michiels, L., Starke, A. D., Tintarev, N., & Vrijenhoek, S. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638319 · Cited by 3 · see citing works
Self-citationLiu, J.-Q., & Azzopardi, L. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661382 · Cited by 8 · see citing works
Allen, G., Hu, A., & Gadiraju, U. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21984 · Cited by 3 · see citing works
Self-citationSleeman, D. H., & Gilhooly, K. J. (2023) · AI Magazine · doi:10.1002/aaai.12135 · Cited by 3 · see citing works
Disentangling Web Search on Debated Topics: A User-Centered Exploration
Rieger, A., Kulane, S., Gadiraju, U., & Pera, M. S. (2024) · User Modeling, Adaptation, and Personalization · doi:10.1145/3627043.3659559 · Cited by 3 · see citing works
Self-citationLiu, J., & He, J. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716453 · Cited by 2 · see citing works
Unveiling Coyote Ads: Detecting Human Smuggling Advertisements on Social Media
Kodandaram, S. R., Sunkara, M., Ferdous, M. J., Poursardar, F., & Ashok, V. G. (2024) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3648188.3675139 · Cited by 2 · see citing works
NORMalize 2024: The Second Workshop on Normative Design and Evaluation of Recommender Systems
Starke, A. D., Vrijenhoek, S., Michiels, L., Kruse, J., & Tintarev, N. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3687103 · Cited by 2 · see citing works
Self-citationTHE RELATIONSHIP BETWEEN INTELLECTUAL EXTREMISM AND SOCIAL VALUES AMONG SAUDI UNIVERSITY STUDENTS
Khasawneh, M. A. S., & Khasawneh, M. A. S. (2023) · Journal of Southwest Jiaotong University · doi:10.35741/issn.0258-2724.58.2.58 · Cited by 2 · see citing works
CrowdDC: Ranking From Crowdsourced Paired Comparison With Divide-and-Conquer
Wang, M.-H., Zhang, C.-Y., & Song, J.-R. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3296632 · Cited by 1 · see citing works
Lyu, H., Bai, Y., Liang, X. T., Das, U., Shi, C., Gong, L., Li, Y., Sun, M., Ge, M., & Ma, X. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645145 · Cited by 1 · see citing works
Schauer, A. M. K., Afolabi, H., & Fu, K. K. (2025) · Design Science · doi:10.1017/dsj.2025.10006 · Cited by 1 · see citing works
Oppenlaender, J., Gadiraju, U., & Hosio, S. J. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737467 · Cited by 1 · see citing works
Self-citationSkeptik: A Hybrid Framework for Combating Potential Misinformation in Journalism
Fan, A., Lei, F., Corman, S. R., & Maciejewski, R. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3766891 · Cited by 1 · see citing works
A Systematic Review on Human Roles, Solutions, and Methodological Approaches to Address Bias in AI
Hashky, A., & Ragan, E. D. (2026) · ACM Computing Surveys · doi:10.1145/3793667 · Cited by 1 · see citing works
Informing Users about Data Imputation: Exploring the Design Space for Dealing With Non-Responses
Bhattacharjee, A., Song, H., Wu, X., Tomlinson, J., Reza, M., Chowdhury, A. E., Deliu, N., Price, T., & Williams, J. J. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27544 · Cited by 1 · see citing works
Mokoena, O. P., Sam, N. T., Ramarumo, T., & Seeletse, S. M. (2023) · International Journal of Entrepreneurship and Sustainability Studies · doi:10.31098/ijeass.v3i2.1972 · Cited by 1 · see citing works
Wang, J., Baharav, C., Shah, N. B., Woolley, A. W., & Ravi, R. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.22000
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_6 · Cited by 1 · see citing works
Bounded Rationality in Decision-Making Under Uncertainty
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_4 · Cited by 1 · see citing works
Back to the Fundamentals: Extend the Rational Assumptions
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_5
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_8
What You Show is What You Get! Gestures for Microtask Crowdsourcing
Allen, G., Hu, A., & Gadiraju, U. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581754.3584175
Self-citationA Crowdsourcing Approach for Identifying Potential Stereotypes in the Collected Data
Christoforou, E., Orphanou, K., Kyriacou, M., & Otterbacher, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-61281-7_1
Leveraging LLMs for Fair Data Labeling and Validation in Crowdsourcing Environments [Vision Paper]
Serouis, I. M., & Sèdes, F. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825193
Allen, G., & Gadiraju, U. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729184
Self-citationWang, D., Shang, L., & Zhang, Y. (2025) · Springer eBooks · doi:10.1007/978-3-031-90080-8_8
Wang, D., Shang, L., & Zhang, Y. (2025) · Springer eBooks · doi:10.1007/978-3-031-90080-8_1
Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali
Reaj, M. A. H., Gupta, R. D., Pritha, J. S., Noman, A. A., Ahmed, A., Mohiuddin, G. M., & Liew, T. H. (2025) · International Conference on Smart Cities, Automation & Intelligent Computing Systems (ICON-SONICS) · doi:10.1109/icon-sonics67145.2025.11309238
Kapoor, S., Jasola, M., & Jasola, A. (2026) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-80592-601-620261007
Human-AI Collaboration in Crowdsourcing: Enhancing Social Computing with Machine Learning
Bansal, S., & Mohan, N. (2025) · Asian Conference on Communication and Networks (ASIANComNet) · doi:10.1109/asiancomnet68615.2025.11579601
Bounded Minds, Generative Machines
Liu, J. (2026) · Communications of the ACM · doi:10.1145/3790104
Meindl, M., & Wilkins, D. (2026) · The British Journal of Social Work · doi:10.1093/bjsw/bcag212
- Authors
- Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N.
- Affiliations
- Delft University of Technology (NL)
- Maastricht University (NL)
- Venue
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
- Published
- 2021-10-04
- Volume
- 9
- Pages
- 48-59
- ISSN
- 2769-1330 (ISSN-L)
- 2769-1349
- OpenAlex
- W3212896246
- Cited by
- 86
- References
- 76
- Citations by year
- 2026: 4 · 2025: 18 · 2024: 19 · 2023: 23 · 2022: 22
- FWCI
- 18.68
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- cognitive biases, crowdsourcing, confirmation bias, anchoring effect, crowdsourced data quality, checklists, task design, data reuse, search results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
