Cognitive Biases in Crowdsourcing
The DOI matches its Crossref record; 54 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [36]
Citations
Cites
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis
Huber, J., Payne, J. W., & Puto, C. (1982) · Journal of Consumer Research · doi:10.1086/208899 · Cited by 1,943 · cited 1×
“The Decoy Effect (sometimes also referred to as the Asymmetric Dominance Effect) occurs when workers’ preference between options A and B changes in favor of option B when an option C is presented, which is similar but clearly inferior to option B [31].”
ResultCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 1×
“An early set of experiments [1, 22, 38, 39] note that aggregated labels of multiple untrained crowd workers can reach a quality comparable to that of a single highly trained NIST assessor.”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Following Dawid and Skene[12],whoinvestigatedisagreementsbetween diagnoses posed by multiple individual medical doctors, there have been several successful attempts at harnessing similar methods for crowdsourcing quality assurance [29, 51].”
Background(2008) · Cited by 1,657
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806 · cited 1×
“In each condition, we train a LambdaMART classifier [56] using a range of standard learning-to-rank features 8 and a neural network-based ranker [24].”
Method
Research works printed but not linked: 10
- Cyril Cleverdon, Readings in Information Retrieval. Readings in Information Retrieval, bibfieldeditor
- Collins-Thompson Kevyn, 2014, TREC 2013 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Collins-Thompson Kevyn, 2015, TREC 2014 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Eickhoff Carsten, 2011, Proceedings of the workshop on crowdsourcing for search and data mining(CSDM) at the fourth ACM international conference on web search and data mining(WSDM). 11--14
- Hirth Matthias, 2010, Cheat-detection mechanisms for crowdsourcing
- Roberts Kirk, 2016, Overview of the TREC 2016 Clinical Decision Support Track. TREC.
- Tversky Amos, Judgment under uncertainty: Heuristics and biases. Utility, probability, and human decision making
- Vuurens Jeroen, 2011, ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR'11)
- Ye Peng, 2013, Conf. on Machine Learning. 1--7.
- Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20. Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20.
Cited by
Items 101–200 of 209
Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs
Siro, C., Aliannejadi, M., & de Rijke, M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657712 · Cited by 9
“Research, such as [4, 16, 38], highlights cognitive biases and cognitive load as significant factors influencing crowdworker judgments, which can skew the evaluations.”
BackgroundOn Annotation Methodologies for Image Search Evaluation
Shao, Y., Liu, Y., Zhang, F., Zhang, M., & Ma, S. (2019) · ACM Transactions on Information Systems · doi:10.1145/3309994 · Cited by 9
“Since cognitive bias in crowsourcing does exist [15], proper aggregation and quality check methods should be considered.”
BackgroundMitigating Bias in Algorithmic Systems -- A Fish-Eye ViewPreprint
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., AlanHartman, & Kuflik, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.16953 · Cited by 9
Bassi, D., Dimitrov, D., D’Auria, B., Alam, F., Hasanain, M., Moro, C., Orrù, L., Turchi, G. P., Nakov, P., & Martino, G. D. S. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.922 · Cited by 8
Hobley, B., Mackiewicz, M., Bremner, J., Dolphin, T. J., & Arosio, R. (2023) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2023.3312820 · Cited by 8
Yang, Y., Zhang, T., & Zhu, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-50670-3_18 · Cited by 8
Vahidnia, M. H. (2024) · Earth Science Informatics · doi:10.1007/s12145-024-01345-z · Cited by 8
“While community contributions offer valuable insights, it is crucial to acknowledge the potential for bias stemming from individual perspectives (Eickhoff 2018; Samulowska et al. 2021).”
User group analytics: hypothesis generation and exploratory analysis of user data
Omidvar-Tehrani, B., Amer-Yahia, S., & Borromeo, R. M. (2018) · The VLDB Journal · doi:10.1007/s00778-018-0527-4 · Cited by 8
“To remove “bias” from our study, we pick a task at random at each iteration of the user study [32].”
MethodErlei, A., Cau, F. M., Georgiev, R., Kumar, S. C., Bizer, K., & Gadiraju, U. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791420 · Cited by 7
“To avoid creating an ambiguity effect as shown in prior literature [44], we removed the Loan-ID attribute, as it provides little information for decision-making—resulting in eleven attributes.”
Contrastive ESA: Human Evaluation of Multiple Translations at Once
Zouhar, V., Grundkiewicz, R., Rajaee, S., Riley, P., Bawden, R., Koehn, P., Carpuat, M., & Kocmi, T. (2026) · Conference on Machine Translation (WMT) · doi:10.18653/2026.wmt-main.47 · Cited by 7
Transparent sociodemographic inference on reddit from self-declarations
Cinus, F., Monti, C., Bajardi, P., & Morales, G. D. F. (2026) · Scientific Reports · doi:10.1038/s41598-026-75155-2 · Cited by 7
Liu, J., & He, J. (2025) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3698204.3716453 · Cited by 7
“[25] has examined cognitive biases (including decoy effect) in Crowdsourcing tasks and measured their effects on user judgments of documents.”
InfluentialTRACE: A Stigmergic Crowdsourcing Platform for Intelligence Analysis
Xia, H., Østerlund, C. S., McKernan, B., Folkestad, J., Rossini, P., Boichak, O., Robinson, J., Kenski, K. M., Myers, R., Clegg, B. A., et al. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.053 · Cited by 7
The original sin of crowd work for human subjects research
Xia, H. (2022) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-12-2021-0126 · Cited by 7
Hube, C., Fetahu, B., & Gadiraju, U. (2018) · Cited by 6
Mitigating Observation Biases in Crowdsourced Label Aggregation
Ueda, R., Takeuchi, K., & Kashima, H. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956439 · Cited by 6
“Eickhoff [18] investigates the impact of several cognitive biases such as the bandwagon effect in crowdsourced experiments, and shows that inappropriate task design leads to poor accuracy.”
BackgroundA Model for Cognitive Personalization of Microtask Design
Paulino, D., Guimarães, D., Correia, A., Ribeiro, J., Barroso, J., & Paredes, H. (2023) · Sensors · doi:10.3390/s23073571 · Cited by 6
“Eickhoff [35] conducted a study to identify the cognitive biases (i.e., systematic errors in the thinking processes) of crowd workers and indicated that microtask design should take these biases into consideration, in order to avoid a decrease in terms of work performance.”
BackgroundMitigating Bias in Algorithmic Systems - A Fish-Eye View
Orphanou, K., Otterbacher, J., Kleanthous, S., Batsuren, K., Giunchiglia, F., Bogina, V., Tal, A. S., Hartman, A., & Kuflik, T. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6240582 · Cited by 6
Chen, N., Fang, H., Liu, J., Wei, W., Sakai, T., & Wu, X.-M. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3779384 · Cited by 5
Yasmin, R., Grassel, J. T., Hassan, M. M., Fuentes, O., & Escobedo, A. R. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18949 · Cited by 5
“More generally, relying on a single form of input elicitation may decrease the quality of the collective decision due to cognitive biases such as anchoring, band-wagon effect, decoy effect, etc. (Eickhoff 2018).”
BackgroundPhishRepo: A Seamless Collection of Phishing Data to Fill a Research Gap in the Phishing Domain
Ariyadasa, S. N., Fernando, S., & Fernando, S. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0130597 · Cited by 5
“From a different perspective, [48] studied the effect of cognitive biases in crowdsourcing.”
BackgroundGörzen, T. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.531 · Cited by 5
“phases, the risk of the anchoring effect occurring is quite high when the information which potentially acts as anchor is revealed to workers in subsequent stages [10].”
Co-destruction Patterns in Crowdsourcing: Formal/Technical Paper
Bidar, R., ter Hofstede, A. H. M., & Sindhgatta, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49435-3_4 · Cited by 5
“When this inclination is based on a contributor’s subjective judgement, it may create a bias which may lead to unfair decisions toward others [13].”
BackgroundFairness and Bias in Truth Discovery Algorithms: An Experimental AnalysisPreprint
Lazier, S., Thirumuruganathan, S., & Anahideh, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.12573 · Cited by 5
“Workers can be imperfect, unmotivated, or influenced by various cognitive biases [10].”
BackgroundSleeman, D. H., & Gilhooly, K. J. (2023) · AI Magazine · doi:10.1002/aaai.12135 · Cited by 5
Blodgett, S. L., Curry, A. C., Dev, S., Madaio, M., Nenkova, A., Yang, D., & Xiao, Z. (2024) · doi:10.18653/v1/2024.hcinlp-1 · Cited by 5
Decoy Effect in Search Interaction: A Pilot StudyPreprint
Chen, N., Liu, J., Sakai, T., & Wu, X.-M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.02362 · Cited by 5
“In the field of information retrieval, Eickhoff [6] examined the impact of a decoy document on thresholds and strategies in crowd-sourcing relevance judgments, showing that assessors could increase the relevance rating of target document when it is shown with the decoy document.”
BackgroundInfluentialDisentangling Web Search on Debated Topics: A User-Centered Exploration
Rieger, A., Kulane, S., Gadiraju, U., & Pera, M. S. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3627043.3659559 · Cited by 4
“Cognitive biases arising from the task design of crowdsourced user studies can negatively impact data quality [21].”
BackgroundEvaluating Cognitive Biases in Conversational and Generative IIR: A Tutorial
Azzopardi, L. A., & Liu, J. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698437 · Cited by 4
Duan, X., Li, Z., Ho, C.-J., & Yin, M. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.119 · Cited by 4
“More recently, it has been recognized that such error can also reflect the “bias” resulted from the annotator’s own cognitive bias (La Barbera et al., 2020; Draws et al., 2022) or the task designs (Eickhoff, 2018; Zhuang et al., 2015).”
Challenges and strategies for running controlled crowdsourcing experiments
Ramírez, J., Báez, M., Casati, F., Cernuzzi, L., & Benatallah, B. (2020) · Latin American Computer Conference (CLEI) · doi:10.1109/clei52000.2020.00036 · Cited by 4
PerPaDa: A Persian Paraphrase Dataset based on Implicit Crowdsourcing Data Collection
Mohtaj, S., Tavakkoli, F., & Asghari, H. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3rnjwejtb2xa · Cited by 4
Does Evidence from Peers Help Crowd Workers in Assessing Truthfulness?
Xu, J., Han, L., Fan, S., Sadiq, S. W., & Demartini, G. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3487553.3524236 · Cited by 4
“Eickhoff [4] leverages human metadata in the crowdsourced relevance assessment task in which peers’ assessments are presented to crowd workers.”
BackgroundCrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designsPreprint
Ramírez, J., Degiacomi, S., Zanella, D., Báez, M., Casati, F., & Benatallah, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02800 · Cited by 4
“…2014), allocated time to completion (Maddalena et al. 2016), pricing, and reward schemes (Difallah et al. 2014), human aspects in collaboration and individual biases (Drapeau et al. 2016; Eickhoff 2018), and even characteristics of the crowd mar-ketplace and work environment (Gadiraju et al. 2017).”
Methods for detecting and mitigating linguistic bias in text corpora
Hube, C. (2020) · Institutional Repository of Leibniz Universität Hannover (Leibniz Universität Hannover) · doi:10.15488/9873 · Cited by 4
Knowledge Graphs Evolution and Preservation -- A Technical Report from ISWS 2019
Abbas, N., Alghamdi, K., Alinam, M., Alloatti, F., Amaral, G., d'Amato, C., Asprino, L., Beno, M., Bensmann, F., Biswas, R., et al. (2020) · arXiv
“• In [51] cognitive bias are considered a source for noisy label submissions, mentioning that people can create subjective social realities.”
BackgroundLiu, J., & Azzopardi, L. A. (2025) · ACM SIGIR Forum · doi:10.1145/3769733.3769736 · Cited by 3
“These include rank position bias (favoring top-ranked results), confirmation bias (seeking information that aligns with prior beliefs), and anchoring (fixating on the first or most salient piece of information) [Wang and Liu, 2024b; White, 2013; Eickhoff, 2018; Ji et al., 2024].”
Bias in, Bias out: Annotation Bias in Multilingual Large Language Models
Cui, X., Huang, Z., & Adel, N. (2025) · doi:10.26615/978-954-452-101-1-001 · Cited by 3
‘Ghostcrafting AI’: Under the Rug of Platform Labor
Rahman, A. T. M. M., & Sultana, S. (2026) · ACM SIGCAS/SIGCHI Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3811242.3819087 · Cited by 3
Paulino, D., Correia, A., Yagui, M. M. M., Barroso, J., Liberato, M. L. R., Vivacqua, A. S., Grover, A., Bigham, J. P., & Paredes, H. (2023) · IEEE Access · doi:10.1109/access.2023.3319597 · Cited by 3
“As the technologies enabling crowdsourcing options have evolved and matured over the years, a body of evidence has accrued indicating that the quality of tasks can significantly decrease when cognitive biases are not taken into account in the task design process [39].”
BackgroundSearch Interface Design and Evaluation
Liu, C., Liu, Y., Liu, J., & Bierig, R. (2021) · doi:10.1561/9781680839234 · Cited by 3
A Domain-Aware Crowdsourcing System with Copier Removal
Fang, X. S., Si, S., Sun, G., Wu, W., Wang, K., & Lv, H. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0416-7_78 · Cited by 3
On the Impact of Showing Evidence from Peers in Crowdsourced Truthfulness Assessments
Xu, J., Han, L., Sadiq, S. W., & Demartini, G. (2023) · ACM Transactions on Information Systems · doi:10.1145/3637872 · Cited by 3
“Among the related literature, the work by Eickhoff [16] focuses on a similar problem.”
BackgroundInfluentialMitigating Label Bias via Decoupled Confident LearningPreprint
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08945 · Cited by 3
“Bias in human-generated labels With the rise of crowd-sourcing services (Howe, 2008), such as Amazon Mechanical Turk, researchers have noted the risks of annotator cognitive biases (Eickhoff, 2018; Draws et al., 2021) and stereo-typing in annotator judgments (Otterbacher, 2015).”
BackgroundBias in, Bias out: Annotation Bias in Multilingual Large Language Models
Cui, X., Huang, Z., & Adel, N. (2025) · e-space (Manchester Metropolitan University) · doi:10.48550/arxiv.2511.14662 · Cited by 2
“Annotator pools often differ demographically from both the dataset’s source community and its intended application domain (Eickhoff, 2018; Bender et al., 2021).”
Modeling Visual Rhetoric and Semantics in Multimedia
Thomas, C. (2020) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 2
Quantifying Worker Reliability for Crowdsensing Applications: Robust Feedback Rating and Convergence
Xie, H. L., & Lui, J. C. S. (2021) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3072477 · Cited by 2
“We model the cognitive bias [9] in task selection via two steps: (1) Model how historical ratings may influence workers’ self-assessment of skills; (2) Model how skill self-assessment influences task selection.”
BackgroundMethodUncovering the Potential of Cognitive Personalization for UI Adaptation in Crowd Work
Paulino, D., Correia, A., Guimarães, D., Barroso, J., & Paredes, H. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776164 · Cited by 2
“…[28]) also explored the use of social networks as a source of information for worker-task matching purposes From a cognitive perspective, Eickhoff [29] stressed the importance of task design and systematically addressed the detrimental effects of four common types of cognitive biases on the…”
BackgroundCognitive Personalization in Microtask Design
Paulino, D., Correia, A., dos Reis, A. M., Guimarães, D., Rudenko, R., Nunes, C., Silva, T., Barroso, J., & Paredes, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05028-2_5 · Cited by 2
Bounded Rationality in Decision-Making Under Uncertainty
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_4 · Cited by 2
An Overview of Computational Approaches for Interpretation AnalysisPreprint
Blandfort, P., Hees, J., & Patton, D. U. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04028 · Cited by 2
“For crowdsourcing, Eickhoff [32] outlines several quality issues and performs dedicated experiments for analyzing cognitive biases of annotators.”
BackgroundAdaptive Crowdsourcing Via Self-Supervised LearningPreprint
Kagrecha, A., Marklund, H., Roy, B. V., Jeon, H. J., & Zeckhauser, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.13239 · Cited by 2
“Third, works (Faltings et al., 2014; Eickhoff, 2018; Geiger et al., 2020) discuss how cognitive biases can negatively affect the quality of crowdsourced data.”
BackgroundUnderstanding Viewpoint Biases in Web Search Results
Draws, T. (2023) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:1b177026-6af7-48f3-ba04-ab7109db3c36 · Cited by 2
Employing Transformers and Humans for Textual-Claim Verification
Saeed, M. (2022) · theses.fr (ABES) · doi:10.70675/3e02a3a6z8281z4d84z8f38ze617eaaf969b · Cited by 2
Zhang, Y., & Cui, J. (2026) · Journal of Documentation · doi:10.1108/jd-11-2025-0343 · Cited by 1
Exploring LLMs for Generating Communicational Actions of External Interfaces on Autonomous Vehicles
Gui, X., Xia, D., Colley, M., Seo, S. H., Chang, C.-M., Javanmardi, E., Tsukada, M., & Igarashi, T. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3810200 · Cited by 1
“They are constrained by designer cognitive biases [21, 58] and cannot possibly account for the combinatorial complexity of dynamic real-world interactions.”
Report on the First INFormation Access in Uncertainty ScEnarios (INFUSE) Workshop at ECIR 2026
Rieger, A., Yu, R., Fard, A. E., Mattis, N. M., Trippas, J. R., Bink, M., Dietze, S., Elsweiler, D., Магдановна, Е. Л., Fuhr, N., et al. (2026) · ACM SIGIR Forum · doi:10.1145/3841298.3841308 · Cited by 1
“Instead, they are interpretations shaped by the assumptions and perceptions of the people who created the labelling schema and who labelled the items of information/ search results [Eickhoff, 2018].”
From Rankings to Ratings: Rank Scoring via Active Learning
Jack, O., Jane, D. S., & Brian, M. N. (2018) · Studies on the semantic web · doi:10.3233/978-1-61499-894-5-69 · Cited by 1
“This idea has had consequences for any field of research in which judgement-based data is collected; and has been applied to sound quality evaluation [22], crowdsourcing [6], and collaborative filtering [1] among others.”
BackgroundPreliminary Experiments to Examine the Stability of Bias-Aware Techniques
Kamishima, T., Akaho, S., Baba, Y., & Kashima, H. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-78818-6_4 · Cited by 1
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_6 · Cited by 1
Implications and New Directions for IR Research and Practices
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_7 · Cited by 1
Mokoena, O. P., Sam, N. T., Ramarumo, T., & Seeletse, S. M. (2023) · International Journal of Entrepreneurship and Sustainability Studies · doi:10.31098/ijeass.v3i2.1972 · Cited by 1
“Different studies profiled frameworks to offset the drawbacks posed by crowdsourcing (Dabiel et al., 2018; Eickhoff, 2018; Draws et al., 2021).”
BackgroundPaulino, D., Ferreira, J. R., Netto, A. T. C., Correia, A., Ribeiro, J., Guimarães, D., Barroso, J., & Paredes, H. (2024) · SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health · doi:10.1109/segah61285.2024.10639597 · Cited by 1
“Concerning the design of tasks in microwork scenarios, a body of research has explored the influence of cognitive bias on changes in the layout of microtasks [4].”
BackgroundSchneider, J., Eisenhardt, D., Utama, C., & Meske, C. (2025) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-80122-8_5 · Cited by 1
Usage of a Cognitive Bias Web-game to Increase Accurate Interpretation of Online Consumer Reviews
Paulino, D., Netto, A. T. C., Guimarães, D., Barroso, J., & Paredes, H. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033636 · Cited by 1
Proceedings of the CSCW 2021 Workshop -- Investigating and Mitigating Biases in Crowdsourced DataPreprint
Hettiachchi, D., Sanderson, M., Gonçalves, J., Hosio, S. J., Kazai, G., Lease, M., Schaekermann, M., & Yılmaz, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14322 · Cited by 1
“For brevity, we illustrate the challenges using examples in the crisis management domain, where filtering social media message streams can be crucial for acquiring situational awareness[7].”
BackgroundMethodInfluentialSamulowska, M., Chmielewski, S., Raczko, E., Lupa, M., Myszkowska, D., & Zagajewski, B. (2020) · Research Square · doi:10.21203/rs.3.rs-61581/v1 · Cited by 1
“…reported subjective data bias at levels between 5% and 35%, depending on the simplicity of the tasks assigned to citizens; Hube et al. (2019) presented data bias at 15–17% in a crowdsourced data set; and Eickhoff (2018) pointed out an accuracy rate reduction by 20% due to cognitive data bias.”
BackgroundMitigating Cognitive Biases in Multi-Criteria Crowd AssessmentPreprint
Ito, S., & Kashima, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.18938 · Cited by 1
“Eickhoff has systematically investigated four types of cognitive effects—ambiguity, anchoring, bandwagon, and decoy effect—based on experimental studies on crowdsourced document relevance assessment tasks (Eickhoff, 2018).”
BackgroundBias-Aware Mislabeling Detection via Decoupled Confident LearningPreprint
Li, Y., De‐Arteaga, M., & Saar‐Tsechansky, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07216 · Cited by 1
“…settings where the true label is unobservable or costly to obtain, and observed labels are instead derived from proxies, human judgments or measurement instruments that can embed label bias due to social structural biases, cognitive heuristics, or gaps in domain knowledge [50, 51, 67, 41, 27, 25].”
Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance AssessmentPreprint
Chen, N., Fang, H., Wang, P., Liu, J., Sakai, T., & Wu, X.-M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.01862 · Cited by 1
“Evaluating document quality, including aspects such as relevance and usefulness, is a cognitively demanding task that is inherently influenced by subjectivity, biases, and mental shortcuts [13–15, 25, 45, 46, 73, 76].”
Deep learning interpretability with visual analytics : Exploring reasoning and bias exploitation
Jaunet, T. (2022) · theses.fr (ABES) · doi:10.70675/a754696bzcd37z49a2zb2a7z2ad1bfd9b620 · Cited by 1
Empirical essays on information asymmetries on digital platforms
Belletti, C. (2024) · theses.fr (ABES) · doi:10.70675/e89a4fc0z1d16z455fz8b32z883c662c960d · Cited by 1
Ochieng', D. M. (2018) · DUT Open Scholar (Durban University of Technology) · doi:10.51415/10321/3205
Computational Approaches to Subjective Interpretation of Multimedia Messages
Blandfort, P. (2020) · Publication Server of Kaiserslautern University of Technology (Kaiserslautern University of Technology)
Pre-screening workers to overcome bias amplification in online labour markets
Vercammen, A., Marcoci, A., & Burgman, M. A. (2021) · PLoS ONE · doi:10.1371/journal.pone.0249051
Leveraging Collaborative Crowdsourcing to Empower Workers to Improve Their Organizations
Mateo, J. C., McCloskey, M. J., Grandjean, A., & Cone, S. M. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-80876-1_43
Cheap IR evaluation: fewer topics, no relevance judgements, and crowdsourced assessments
Roitero, K. (2020) · ACM SIGIR Forum · doi:10.1145/3483382.3483400
“Eickhoff [49] looked at the effect of cognitive biases in crowdsourced relevance judgement tasks.”
BackgroundLiu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_1
From Rational Agent to Human with Bounded Rationality
Liu, J. (2023) · The information retrieval series · doi:10.1007/978-3-031-23229-9_3
Key-phrase boosted unsupervised summary generation for FinTech organizationPreprint
Deshpande, A., Goyal, S., Nagwanshi, P., & Tripathy, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10294
“Crowd-sourcing comes with its own set of problems, as it is often expensive and presents problems with annotator bias [10].”
BackgroundAssessment of bias in a crowdsourcing accessibility system using serious games
Barguil, J. M. M. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2023.tde-02012024-175653
Genetically-diverse crowds are wiser
Barneron, M. S., Yaniv, I., Abramson, L., & Knafo‐Noam, A. (2024) · Personality and Individual Differences · doi:10.1016/j.paid.2024.112823
Video Label Refinement for Temporal Localization
Piane, J., Ramaraj, T., Furst, J. D., & Raicu, D. S. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209895
“Eickhoff used three public datasets for information retrieval tasks to evaluate crowdsourcing class labels and found that the way annotation tools provide information to the annotator can impact the resulting labels [6].”
Paakki, H., Toivanen, P., & Kajava, K. (2025) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2025.5980
In Crowd Veritas: Leveraging Human Intelligence To Fight MisinformationPreprint
Soprano, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.09221
Exploring algorithmic realism in the data economy
Barbosa, N. M. (2021) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Understanding and Addressing Uncertainty of the Crowd
Chen, Q. Z. (2022) · ResearchWorks at the University of Washington (University of Washington)
A human-in-the-loop framework to handle implicit bias in crowdsourced KGs
Tirado, A. C. M., Oelen, A., Pasqual, V., Shi, M., Umbrico, A., Xu, W., & Celino, I. (2020) · Open Research Online - ORO (The Open University)
Understanding Context Effects in the Evaluation of Music Similarity
van den Berg, M. (. (2021) · Research Repository (Delft University of Technology)
Aggregating Crowd of LLMs for Cost-Effective Data Annotation
Liu, J., & Hou, X. (2026) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2026.findings-eacl.125
Tomás, D. B. (2025) · DIGITAL.CSIC (Spanish National Research Council (CSIC))
Lu, Y.-C. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799172
“Prior work shows that such judgments are systematically shaped by cognitive biases and heuristics, which can produce skewed or imbalanced expectations [3, 7, 16, 21, 24, 25].”
Towards citizen engagement: the emergence of crowdsourcing
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00007
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.fm6
Theoretical foundations: fundamentals, process, and principles of crowdsourcing in the public sector
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00008
Case study: crowd wisdom, crowd creation, crowd voting
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00010
Towards more effective management: new trends in crowdsourcing in the public sector
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00011
The road forward: future of crowdsourcing in the public sector
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00012
Digital Citizens and Crowdsourcing: Managing Engagement in the Public Sector
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509
Step by step: implementing and managing crowdsourcing in the public sector
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.00009
Metadata
- Authors
- Eickhoff, C.
- Affiliations
- ETH Zurich (CH)
- Corresponding author
- Eickhoff, Carsten
- ORCID
- Eickhoff, Carsten
- Venue
- Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2018-02-02
- Pages
- 162-170
- OpenAlex
- W2783668259
- Cited by
- 144
- References
- 58
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 21 · 2023: 24 · 2022: 24 · 2021: 15 · 2020: 16 · 2019: 20 · 2018: 8
- FWCI
- 16.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- crowdsourcing, cognitive biases, quality control, annotation, data curation, information retrieval evaluation, cheating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.