Cognitive Biases in Crowdsourcing
The DOI matches its Crossref record; 54 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [36]
Citations
Cites
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis
Huber, J., Payne, J. W., & Puto, C. (1982) · Journal of Consumer Research · doi:10.1086/208899 · Cited by 1,943 · cited 1×
“The Decoy Effect (sometimes also referred to as the Asymmetric Dominance Effect) occurs when workers’ preference between options A and B changes in favor of option B when an option C is presented, which is similar but clearly inferior to option B [31].”
ResultCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 1×
“An early set of experiments [1, 22, 38, 39] note that aggregated labels of multiple untrained crowd workers can reach a quality comparable to that of a single highly trained NIST assessor.”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Following Dawid and Skene[12],whoinvestigatedisagreementsbetween diagnoses posed by multiple individual medical doctors, there have been several successful attempts at harnessing similar methods for crowdsourcing quality assurance [29, 51].”
Background(2008) · Cited by 1,657
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806 · cited 1×
“In each condition, we train a LambdaMART classifier [56] using a range of standard learning-to-rank features 8 and a neural network-based ranker [24].”
Method
Research works printed but not linked: 10
- Cyril Cleverdon, Readings in Information Retrieval. Readings in Information Retrieval, bibfieldeditor
- Collins-Thompson Kevyn, 2014, TREC 2013 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Collins-Thompson Kevyn, 2015, TREC 2014 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Eickhoff Carsten, 2011, Proceedings of the workshop on crowdsourcing for search and data mining(CSDM) at the fourth ACM international conference on web search and data mining(WSDM). 11--14
- Hirth Matthias, 2010, Cheat-detection mechanisms for crowdsourcing
- Roberts Kirk, 2016, Overview of the TREC 2016 Clinical Decision Support Track. TREC.
- Tversky Amos, Judgment under uncertainty: Heuristics and biases. Utility, probability, and human decision making
- Vuurens Jeroen, 2011, ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR'11)
- Ye Peng, 2013, Conf. on Machine Learning. 1--7.
- Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20. Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20.
Cited by
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
BackgroundCulpepper, J. S., Diaz, F., & Smucker, M. D. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274788 · Cited by 178
Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval
Azzopardi, L. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446023 · Cited by 176
“between result items (where one is relevant, and one is not-relevant), the introduction of another non-relevant item which is similar but dominated by the other non-relevant item may deceive searchers into selecting a non-relevant item [18].”
BackgroundResultInfluentialUnderstanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129
“Eickhoff studied the prevalence of cognitive biases (ambiguity effect, anchoring, bandwagon and decoy effect) as a source of noise in crowdsourced data curation, annotation and evaluation [9].”
BackgroundStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
“Some of this work argues, for example, that data workers’ cognitive biases [30], their own preferences [81], and political stances [93] can negatively affect data.”
BackgroundRehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning
Barbosa, N. M., & Chen, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300773 · Cited by 89
“In regards to biases originating from crowd work, several prior works have looked at biases introduced by the process of labeling via crowds (e.g., [3, 5, 6, 8, 9, 23, 24, 26, 34]).”
BackgroundToward a Perspectivist Turn in Ground Truthing for Predictive Computing
Cabitza, F., Campagner, A., & Basile, V. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25840 · Cited by 114
A Survey on Task Assignment in Crowdsourcing
Hettiachchi, D., Kostakos, V., & Gonçalves, J. (2022) · ACM Computing Surveys · doi:10.1145/3494522 · Cited by 104
“• Cognitive Biases: The study by Eickhoff [40] investigates cognitive biases and shows that cognitive biases negatively impact crowd task performance in relevance labelling.”
BackgroundCrowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions
Aroyo, L., Dixon, L., Thain, N., Redfield, O., & Rosen, R. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317083 · Cited by 62
“[12] exemplifies the "significant detrimental effects" of cognitive biases on the annotation quality in relevance judgment tasks, when such biases are not taken into consideration in the design of the task.”
BackgroundDon’t Blame the Annotator: Bias Already Starts in the Annotation Instructions
Parmar, M., Mishra, S. R., Geva, M., & Baral, C. R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.130 · Cited by 95
Elangovan, A., Liu, L., Xu, L., Bodapati, S., & Roth, D. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.63 · Cited by 90
“There are several studies on the effects of cognitive biases on information search and retrieval (Lau and Coiera, 2007; White, 2013) and crowd-sourcing (Eickhoff, 2018; Santhanam et al., 2020).”
Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects
Grand, G., & Belinkov, Y. (2019) · Proceedings of the Second Workshop on Shortcomings in Vision and Language · doi:10.18653/v1/w19-1801 · Cited by 76
“any benchmark that involves crowdsourced data is likely to encode certain cognitive and/or social biases (van Miltenburg, 2016; Misra et al., 2016; Eickhoff, 2018).”
BackgroundA Checklist to Combat Cognitive Biases in Crowdsourcing
Draws, T., Rieger, A., Inel, O., Gadiraju, U., & Tintarev, N. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18939 · Cited by 86
“For instance, Eickhoff (2018) showed –through the lens of a standard relevance judgment task– how requesters may deal with biases related to groupthink , anchoring , and the halo effect .”
BackgroundMethodInfluentialCrowdsourcing Geospatial Data for Earth and Human Observations: A Review
Huang, X., Wang, S., Yang, D., Hu, T., Chen, M., Zhang, M., Zhang, G., Biljecki, F., Lu, T., Zou, L., et al. (2024) · Journal of Remote Sensing · doi:10.34133/remotesensing.0105 · Cited by 86
“…fourth, encouraging a broad range of contributors with different demographic backgrounds from different regions for inclusivity and diversity [220,227]; finally, implementing quality control measures during the crowdsourcing process can help identify and filter out biased responses [225,228].”
BackgroundCognitive Biases in Fact-Checking and Their Countermeasures: A Review
Soprano, M., Roitero, K., Barbera, D. L., Ceolin, D., Spina, D., Demartini, G., & Mizzaro, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103672 · Cited by 86
Mitigating Bias in Algorithmic Systems—A Fish-eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2023) · ACM Computing Surveys · doi:10.1145/3527152 · Cited by 52
All Those Wasted Hours: On Task Abandonment in Crowdsourcing
Han, L., Roitero, K., Gadiraju, U., Sarasua, C., Checco, A., Maddalena, E., & Demartini, G. (2019) · Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining · doi:10.1145/3289600.3291035 · Cited by 55
“For example, Ipeirotis et al. [21] provided a solution to distinguish true errors from individual’s systematic biases and Eickhoff [12] looked at the effect of cognitive biases in the crowd on IR evaluation.”
BackgroundThe Impact of Task Abandonment in Crowdsourcing
Han, L., Roitero, K., Gadiraju, U., Sarasua, C., Checco, A., Maddalena, E., & Demartini, G. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2948168 · Cited by 60
“For example, Ipeirotis et al. [15] provided a solu-tion to distinguish true errors from individual’s systematic biases and Eickhoff [16] looked at the effect of cognitive biases in the crowd on IR evaluation.”
BackgroundKnowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective
Zhang, J. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105434 · Cited by 66
“This cognitive bias was further confirmed by successive studies [59], [60].”
ResultRoitero, K., Soprano, M., Fan, S., Spina, D., Mizzaro, S., & Demartini, G. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401112 · Cited by 35
“In the context of crowdsourced relevance judgments, Eickhoff [11] showed how common types of bias can impact the collected judgments and the results of Information Retrieval (IR) evaluation initiatives.”
Davani, A. M., Díaz, M., Baker, D. K., & Prabhakaran, V. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1029 · Cited by 61
Chen, N., Liu, J., Dong, X., Liu, Q., Sakai, T., & Wu, X.-M. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698420 · Cited by 51
“For instance, how the recency effect and the reference-dependent effect influence users’ perceptions of search satisfaction [32, 37]; and how the anchoring effect, decoy effect, expectation confirmation, and priming effect influence users’ judgments of document relevance [13, 46, 48, 57, 58].”
Hettiachchi, D., van Berkel, N., Kostakos, V., & Gonçalves, J. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415181 · Cited by 49
“Previous work by Eickhoff [20] and Alagarai Sampath et al. [1] indicate the possibility of using cognitive tests for crowdsourcing task assignment.”
BackgroundCognitive Biases in Large Language Models: A Survey and Mitigation Experiments
Sumita, Y., Takeuchi, K., & Kashima, H. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707812 · Cited by 47
“We apply the mitigation methods in crowdsourcing to the prompts of LLMs. Cognitive bias also degrades annotation quality in crowd-sourcing [6], and two methods have been proposed to mitigate biases by changing instructions [9]: SoPro and AwaRe.”
Multi-Label Inference for Crowdsourcing
Zhang, J., & Wu, X. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219958 · Cited by 48
“This phenomenon has been extensively studied in recent years [7, 11, 36].”
BackgroundAlonso, O. (2019) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s00904ed1v01y201903icr066 · Cited by 40
Human-Machine Collaboration for Smart Decision Making: Current Trends and Future Opportunities
Geng, B., & Varshney, P. K. (2022) · IEEE International Conference on Collaboration and Internet Computing (CIC) · doi:10.1109/cic56439.2022.00019 · Cited by 40
“Other works show that relevance judgments can be affected by displaying other crowd workers’ judgments (i.e., the bandwagon effect, this is also referred to as Group Think or Herd Behavior) [39], [40].”
Uncovering labeler bias in machine learning annotation tasks
Haliburton, L., Leusmann, J., Welsch, R., Ghebremedhin, S., Isaakidis, P., Schmidt, A., & Mayer, S. (2024) · AI and Ethics · doi:10.1007/s43681-024-00572-w · Cited by 39
“However, datasets embed the cognitive biases of crowd workers [1], which can propagate through machine learning (ML) systems [2].”
BackgroundSearch Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000073 · Cited by 39
Metrology for AI: From Benchmarks to InstrumentsPreprint
Welty, C., Paritosh, P. K., & Aroyo, L. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01875 · Cited by 36
“To ensure quality of the resulted crowdsourced data, various mechanisms have been developed to measure the quality of the raters; (Joglekar, Garcia-Molina, and Parameswaran 2015) provide concise confidence intervals on raters error rates, (Eickhoff 2018) focus on cognitive biases in crowdsourcing, (Ipeirotis 2011) off…”
BackgroundOn Fine-Grained Relevance Scales
Roitero, K., Maddalena, E., Demartini, G., & Mizzaro, S. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210052 · Cited by 35
“Eickhoff [5] looked at the effect of cognitive biases in crowd-sourced relevance judgment tasks.”
BackgroundAlonso, O. (2019) · Synthesis lectures on information concepts, retrieval, and services · doi:10.1007/978-3-031-02318-7 · Cited by 34
Implicit Bias in Crowdsourced Knowledge Graphs
Demartini, G. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317307 · Cited by 32
“Eickhoff [7] looked at the effect of cognitive bias in crowdsourcing.”
BackgroundModelling Cognitive Bias in Crowdsourcing Systems
Saab, F., Elhajj, I. H., Kayssi, A., & Chehab, A. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.04.004 · Cited by 32
VisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30
“On the contrary, VQAv2 [24] dataset is crowdsourced, which lead to more natural questions, but includes cognitive and/or social biases [19] as well as annotation mistakes.”
BackgroundQuality aspects of annotated data: A research synthesis
Beck, J. (2023) · AStA Wirtschafts- und Sozialstatistisches Archiv · doi:10.1007/s11943-023-00332-y · Cited by 29
“…theoretical background often stems from social psychology, have previously been examined, such as anchoring (disproportionate focus on one piece of information) or confirmation bias (tendency to perceive information in a way that con-firms previously held beliefs) (Eickhoff 2018; Hube et al. 2019).”
Accounting for Confirmation Bias in Crowdsourced Label Aggregation
Gemalmaz, M. A., & Yin, M. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/238 · Cited by 29
“The design of crowdsourcing tasks (e.g., what information is shown to workers in what order) may also have subtle impact on workers and trigger their cognitive biases such as the anchoring bias and ambiguity effect [ Eickhoff, 2018; Zhuang et al. , 2015].”
BackgroundRobust Reinforcement Learning from Corrupted Human Feedback
Bukharin, A., Hong, I., Jiang, H., Li, Z., Zhang, Q., Zhang, Z., & Zhao, T. (2024) · neural information processing systems · doi:10.52202/079017-3942 · Cited by 28
Investigating and Mitigating Biases in Crowdsourced Data
Hettiachchi, D., Sanderson, M., Gonçalves, J., Hosio, S. J., Kazai, G., Lease, M., Schaekermann, M., & Yılmaz, E. (2021) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3462204.3481729 · Cited by 27
“While quality assurance methods are commonly used with crowd-sourced data labelling, studies have shown that annotator biases often creep into labelling decisions [6, 7, 18, 31, 34].”
BackgroundMethodToward Cranfield-inspired reusability assessment in interactive information retrieval evaluation
Liu, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103007 · Cited by 27
Toward A Two-Sided Fairness Framework in Search and Recommendation
Liu, J. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578332 · Cited by 27
“In IR and Crowdsourcing labeling, Eickhoff [25] examined the impact of decoy document on users’ thresholds and strategies in standard relevance judgments.”
BackgroundCollect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
Inel, O., Draws, T., & Aroyo, L. M. (2023) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v11i1.27547 · Cited by 25
Blind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLP
Gautam, S., & Srinath, M. (2024) · Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) · doi:10.18653/v1/2024.hcinlp-1.8 · Cited by 25
A method for Smart Idea Allocation in crowd-based idea selection
Banken, V., Ilmer, Q., Seeber, I., & Haeussler, S. (2019) · Decision Support Systems · doi:10.1016/j.dss.2019.113072 · Cited by 24
The Effects of Crowd Worker Biases in Fact-Checking Tasks
Draws, T., La Barbera, D., Soprano, M., Roitero, K., Ceolin, D., Checco, A., & Mizzaro, S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534629 · Cited by 23
“However, because crowdsourcing often relies on contributions from large groups of laypeople with different backgrounds, expertise, and skills, systematic biases among those workers may reduce the quality of their annotations [9, 11, 21].”
BackgroundMethodConstructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model
Chen, N., Zhang, F., & Sakai, T. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531953 · Cited by 23
“To date, a range of studies in IR discipline have shed light on cognitive biases in search interactions and crowdscourcing [9, 12, 15], like the recency effect [18], the ordering effect [8, 25], and the reference dependence effect [17].”
BackgroundLanguré, A. D. L., & Zareei, M. (2023) · IEEE Access · doi:10.1109/access.2023.3331323 · Cited by 23
“It has also been documented that human-labeled resources lack consistency in representing and analyzing emotions across different datasets, tasks, and applications [46].”
BackgroundObjective Assessment of Subjective Tasks in Crowdsourcing Applications.
Haralabopoulos, G., Tsikandilakis, M., Torres, M. T., & McAuley, D. (2020) · Repository@Nottingham (University of Nottingham) · Cited by 23
Predicting Visual Political Bias Using Webly Supervised Data and an Auxiliary Task
Thomas, C., & Kovashka, A. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01506-3 · Cited by 23
“Other works (Sen et al., 2015; Olteanu et al., 2019; Nguyen et al., 2014; Eickhoff, 2018) have analyzed the bias inherent in human annotated data introduced by crowdsourcing annotations.”
BackgroundThe Crowd is Made of People: Observations from Large-Scale Crowd Labelling
Thomas, P., Kazai, G., White, R. W., & Craswell, N. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505815 · Cited by 22
“Past work has demonstrated relevance labels can be affected by cognitive biases or style [17, 30, 53, 55], work patterns [28], or existing beliefs [32], but this work has typically been at relatively small scale; or has induced the effect in question, not observed it “in the wild”.”
BackgroundMethodam-ELO: A Stable Framework for Arena-based LLM EvaluationPreprint
Liu, Z., Li, J., Zhuang, Y., Liu, Q., Shen, S., Ouyang, J., Cheng, M., & Wang, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.03475 · Cited by 22
“Ignoring this variability introduces biases and instability into the evaluation process, further undermining the credibility of both the results and the decisions derived from them (Eickhoff, 2018).”
Han, K., Rezapour, R., Nakamura, K., Devkota, D., Miller, D. C., & Diesner, J. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24714 · Cited by 21
“However, due to concerns about the reliability of annotations provided by non-experts on crowdsourcing platforms (Dow et al., 2012; Eickhoff, 2018), this solution is mainly suitable for straightforward annotation tasks that require little to no domain expertise or specialized training.”
BackgroundComprehensive guidelines for emotion annotation
Islam, M. A., Mukta, M. S. H., Olivier, P. L., & Rahman, M. M. (2022) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3514197.3549640 · Cited by 21
“Specifically, human annotations can be significantly affected by cognitive biases [9].”
BackgroundMulti-Label Truth Inference for Crowdsourcing Using Mixture Models
Zhang, J., & Wu, X. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2951668 · Cited by 21
“Many studies keep focusing on the biased annotation issue in recent years, from the previous inference models [27], [32] to the most recently cognitive mechanism [48].”
BackgroundRoitero, K., Soprano, M., Fan, S., Spina, D., Mizzaro, S., & Demartini, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06915 · Cited by 21
“In the context of crowdsourced relevance judgments, Eickhoff [11] showed how common types of bias can impact the collected judgments and the results of Information Retrieval (IR) evaluation initiatives.”
Robust Link Prediction over Noisy Hyper-Relational Knowledge Graphs via Active Learning
Yu, W., Yang, J., & Yang, D. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645686 · Cited by 20
“A slighted related line of work addresses the human side issues in crowdsourcing, for example, various human biases affected by cognitive [11], cultural and demographic [10] factors, which may lead to sub-optimal performance.”
BackgroundEffect of Cognitive Abilities on Crowdsourcing Task Performance
Hettiachchi, D., van Berkel, N., Hosio, S. J., Kostakos, V., & Gonçalves, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29381-9_28 · Cited by 19
“Eickhoff [15] examined the effect of cognitive biases in crowdsourced relevance labelling tasks and reported that biases could significantly deteriorate the quality of output.”
Samulowska, M., Chmielewski, S., Raczko, E., Lupa, M., Myszkowska, D., & Zagajewski, B. (2021) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi10020046 · Cited by 19
“Kosmala et al. [20] reported subjective data bias at levels between 5% and 35%, depending on the simplicity of the tasks assigned to citizens, Hube et al. [107] presented data bias at 15–17% in a crowdsourced data set, and Eickhoff [108] pointed out an accuracy rate reduction of 20% due to cognitive data bias.”
BackgroundA hybrid human–AI tool for scientometric analysis
Correia, A., Grover, A., Jameel, S., Schneider, D., Antunes, P. A., & Fonseca, B. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10548-7 · Cited by 18
“The latter is assumed to occur in different ways in crowdsourcing, such as when workers allow themselves to be led by the crowd’s behavior instead of their own way of thinking (Eickhoff 2018).”
Wang, B., & Liu, J. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672520 · Cited by 18
“IR researchers have investigated these biases to understand seemingly irrational decisions in users’ search activities and perceptions of search results [e.g. 3, 19, 36].”
Human biases in body measurement estimation
Martynov, K., Garimella, K. V., & West, R. (2020) · EPJ Data Science · doi:10.1140/epjds/s13688-020-00250-x · Cited by 18
Mapping Points of Interest Through Street View Imagery and Paid Crowdsourcing
Maddalena, E., Ibáñez, L., & Simperl, E. P. B. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3403931 · Cited by 18
“This type of bias has been reported in the crowdsourcing literature for text annotation tasks [Eickhoff 2018].”
BackgroundHelping Crisis Responders Find the Informative Needle in the Tweet HaystackPreprint
Derczynski, L., Meesters, K., Bontcheva, K., & Maynard, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09633 · Cited by 18
“Because we crowdsource the work to human annotators, we were mindful of how task design and framing could affect results: taking care of cognitive biases is paramount to achieving good results (Sabou et al. 2014; Derczynski, Bontcheva, et al. 2016; Eickhoff 2018).”
BackgroundSelf-Adaptive Cognitive Debiasing for Large Language Models in Decision-MakingPreprint
Lyu, Y., ren, S. D., Feng, Y., Wang, Z., Chen, Z., Ren, Z., & de Rijke, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.04141 · Cited by 18
“Such studies span retrieval systems [3, 7, 18–20, 31, 37, 57, 65, 67, 78, 79, 87, , Vol. 1, No. 1, Article .”
Zhou, X., Wang, X., & Liu, X. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.911850 · Cited by 17
“…(occurrence per 1,000 words) 1 Simple Text I 4.692 19.231 2 Complex Text II 5.955 12.195 Frontiers in Psychology 06 frontiersin.org revealed that even precise guidelines were given, cognitive bias and disagreement might still occur during the assessment process (Eickhoff, 2018; Islam et al., 2022).”
BackgroundInfluentialYi, F., Singh, A., & Tao, Z. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000101 · Cited by 17
Search Result Explanations Improve Efficiency and Trust
Ramos, J., & Eickhoff, C. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401279 · Cited by 17
“Machine-learned models that use large-scale datasets [8] as a basis for learning may end up mirroring or reinforcing existing bias in the data [7].”
BackgroundSelf-citationWhen More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21994 · Cited by 17
de Groot, E. C. S., & Gadiraju, U. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642429 · Cited by 16
Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System Vulnerability
Chen, N., Liu, J., Fang, H., Luo, Y., Sakai, T., & Wu, X.-M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3708884 · Cited by 16
The Influences of Task Design on Crowdsourced Judgement: A Case Study of Recidivism Risk Evaluation
Duan, X., Ho, C.-J., & Yin, M. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512239 · Cited by 16
“example, a few studies have examined the types of biases that may exist in crowdsourced datasets [22, 44, 64].”
BackgroundDetecting cognitive bias in a relevance assessment task using an eye tracker
Harris, C. G. (2019) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/3314111.3319824 · Cited by 16
“Several cognitive biases, including the anchoring effect, were examined in [Eickhoff, 2018], but the focus of this research was on reducing the noisy inputs of crowdworkers.”
BackgroundVATAS: An Open-Source Web Platform for Visual and Textual Analysis of Social Media
Patton, D. U., Blandfort, P., Frey, W. R., Schifanella, R., McGregor, K. A., & Chang, S. (2020) · Journal of the Society for Social Work and Research · doi:10.1086/707667 · Cited by 16
“It is important to note that unquali fi ed workers are, despite their best efforts, unable to produce an acceptable annotation quality (Eickhoff, 2018).”
BackgroundOn the Role of Large Language Models in Crowdsourcing Misinformation Assessment
Xu, J., Han, L., Sadiq, S. W., & Demartini, G. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31417 · Cited by 14
“This influence can be classified as a bandwagon effect, similar to what has been observed in other crowdsourcing tasks involving advice from humans (Eickhoff 2018).”
BackgroundResultInvestigating Labeler Bias in Face Annotation for Machine Learning
Haliburton, L., Ghebremedhin, S., Welsch, R., Schmidt, A., & Mayer, S. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240191 · Cited by 14
“One under-investigated contributing factor to biased AI tools is labeler bias, which results from cognitive biases [2] in crowd workers and other dynamics in the labeling process [3].”
BackgroundAssessing the potential of generative agents in crowdsourced fact-checking
Costabile, L., Orlando, G. M., Gatta, V. L., & Moscato, V. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100326 · Cited by 14
“Notably, the complexity and domain-specificity of claims—such as those related to political discourse—can significantly affect the accuracy of crowd assessments [24, 25, 26].”
TaskLint: Automated Detection of Ambiguities in Task Instructions
Manam, V. K. C., Thomas, J. D., & Quinn, A. J. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21996 · Cited by 14
“Workers’ individual characteristics affect result quality by way of cognitive factors (Alagarai Sampath, Ra-jeshuni, and Indurkhya 2014), attention (G¨oritz, Borchert, and Hirth 2021; Rothwell et al. 2016), mood (Gadiraju and Demartini 2019; Zhuang and Gadiraju 2019; Xu, Zhou, and Gadiraju 2019; Qiu, Gadiraju, and Boz…”
BackgroundŁajewska, W., & Balog, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615132 · Cited by 14
“The actual quality of the results is influenced by the workers, software platform [30], task design [11], and quality measures employed [9].”
BackgroundEnhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs
Kemmer, R., Yoo, Y., Escobedo, A. R., & Maciejewski, R. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7465 · Cited by 14
“Cognitive biases such as anchoring effect, bandwagon effect, and decoy effect have been found to be present while collecting data in crowdsourced tasks, and negatively affect results (Eickhoff 2018).”
BackgroundCognitive personalization for online microtask labor platforms: A systematic literature review
Paulino, D., Correia, A., Barroso, J., & Paredes, H. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09383-w · Cited by 14
“Another set of significant contributions regards the prevalence of cognitive bias in crowd workers, specifically in the case of document relevance assessment tasks (Eickhoff 2018).”
InfluentialDigital Labor: Challenges, Ethical Insights, and ImplicationsPreprint
Rahman, A. M., & Sultana, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.11788 · Cited by 14
“Some workers receive fewer tasks because of where they live or how their profiles are ranked [37, 151].”
Identifying Algorithmic Decision Subjects' Needs for Meaningful Contestability
Yurrita, M., Verma, H., Balayn, A., Alfrink, K., Gadiraju, U., & Bozzon, A. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757415 · Cited by 12
Detecting and preventing confused labels in crowdsourced data
Krivosheev, E., Bykau, S., Casati, F., & Prabhakar, S. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3407790.3407842 · Cited by 12
Washington, P. Y., Mutlu, O. C., Kline, A., Paskov, K. M., Stockham, N., Chrisman, B., Deveau, N., Surhabi, M., Haber, N., & Wall, D. P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11197 · Cited by 12
“Crowd workers are humans with their own preconceptions and predispositions, and these cognitive biases have been documented in crowdsourcing contexts [58].”
BackgroundResponsible Opinion Formation on Debated Topics in Web Search
Rieger, A., Draws, T., Mattis, N. M., Maxwell, D., Elsweiler, D., Gadiraju, U., McKay, D., Bozzon, A., & Pera, M. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56066-8_32 · Cited by 12
Trustworthy human computation: a survey
Kashima, H., Oyama, S., Arai, H., & Mori, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10974-1 · Cited by 11
“Eickhoff [59] demonstrated the existence of various cognitive biases (ambiguity effect, anchoring, bandwagon effect, and decoy effect) in crowdsourcing”
InfluentialSelecting Workers Wisely for Crowdsourcing When Copiers and Domain Experts Co-exist
Fang, X. S., Si, S., Sun, G., Sheng, Q. Z., Wu, W., Wang, K., & Lv, H. (2022) · Future Internet · doi:10.3390/fi14020037 · Cited by 11
In her Shoes: Gendered Labelling in Crowdsourced Safety Perceptions Data from India
Sengupta, N., Vaidya, A., & Evans, J. A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593987 · Cited by 11
“We demonstrate that there is an urgent need to analyse potential bias in crowdsourced data [10], particularly from the perspective of the Global South, which has received woefully less attention in the scientific discourse around algorithmic fairness [25].”
BackgroundLiu, J., & Azzopardi, L. A. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3661382 · Cited by 11
Siro, C., Aliannejadi, M., & de Rijke, M. (2024) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2024.findings-naacl.80 · Cited by 11
Herd Accountability of Privacy-Preserving Algorithms: A Stackelberg Game Approach
Yang, Y.-T. C., Zhang, T., & Zhu, Q. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3540357 · Cited by 11
“The disparity in the capability of herd behaviors has been extensively studied in collective intelligence [9], [29], [30], [31].”
Yasmin, R., Hassan, M. M., Grassel, J. T., Bhogaraju, H., Escobedo, A. R., & Fuentes, O. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.848056 · Cited by 11
“Relying on a single form of input elicitation may decrease the quality of the collective judgment due to cognitive biases such as anchoring, bandwagon effect, decoy effect, etc. (Eickhoff, 2018).”
BackgroundReliability of crowdsourcing as a method for collecting emotions labels on pictures
Korovina, O., Báez, M., & Casati, F. (2019) · BMC Research Notes · doi:10.1186/s13104-019-4764-4 · Cited by 11
Managing Bias in Human-Annotated Data: Moving Beyond Bias RemovalPreprint
Demartini, G., Roitero, K., & Mizzaro, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13504 · Cited by 11
“To address these issues, the study of bias in data and algorithms gained popularity [1, 12, 16, 19, 21, 23], both in disciplines that study fully-automatic systems such as machine and deep learning [14, 15] and in those that study and develop crowdsourcing-based or hybrid human-in-the-loop systems [4, 5, 7, 8].”
Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking
Roitero, K., Wright, D., Soprano, M., Augenstein, I., & Mizzaro, S. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729960 · Cited by 10
“While crowdsourcing holds significant potential, it is not without challenges [17, 25].”
Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models
Malaviya, C., Bhatia, S., & Yatskar, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.438 · Cited by 10
Azzopardi, L. A., & Liu, J. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638297 · Cited by 10
“• Evaluating and assessing results (order, bandwagon and de-coy effects [17] and reference dependence when judging and rating results and systems [15, 44]).”
BackgroundLiu, J., & He, J. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3820166 · Cited by 10
“Our experiment was approved by the University of Oklahoma Institutional Review Board ( IRB # : 13527).”
On the role of human and machine metadata in relevance judgment tasks
Xu, J., Han, L., Sadiq, S. W., & Demartini, G. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103177 · Cited by 10
A Systematic Literature Survey of Crowdsourcing: Current Status and Future Perspectives
Suyal, H., & Singh, A. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70037 · Cited by 9
Metadata
- Authors
- Eickhoff, C.
- Affiliations
- ETH Zurich (CH)
- Corresponding author
- Eickhoff, Carsten
- ORCID
- Eickhoff, Carsten
- Venue
- Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2018-02-02
- Pages
- 162-170
- OpenAlex
- W2783668259
- Cited by
- 144
- References
- 58
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 21 · 2023: 24 · 2022: 24 · 2021: 15 · 2020: 16 · 2019: 20 · 2018: 8
- FWCI
- 16.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- crowdsourcing, cognitive biases, quality control, annotation, data curation, information retrieval evaluation, cheating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.