Cognitive Biases in Crowdsourcing
The DOI matches its Crossref record; 54 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [36]
Citations
Cites
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis
Huber, J., Payne, J. W., & Puto, C. (1982) · Journal of Consumer Research · doi:10.1086/208899 · Cited by 1,943 · cited 1×
“The Decoy Effect (sometimes also referred to as the Asymmetric Dominance Effect) occurs when workers’ preference between options A and B changes in favor of option B when an option C is presented, which is similar but clearly inferior to option B [31].”
ResultCrowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923 · cited 1×
“An early set of experiments [1, 22, 38, 39] note that aggregated labels of multiple untrained crowd workers can reach a quality comparable to that of a single highly trained NIST assessor.”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Following Dawid and Skene[12],whoinvestigatedisagreementsbetween diagnoses posed by multiple individual medical doctors, there have been several successful attempts at harnessing similar methods for crowdsourcing quality assurance [29, 51].”
Background(2008) · Cited by 1,657
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806 · cited 1×
“In each condition, we train a LambdaMART classifier [56] using a range of standard learning-to-rank features 8 and a neural network-based ranker [24].”
Method
Research works printed but not linked: 10
- Cyril Cleverdon, Readings in Information Retrieval. Readings in Information Retrieval, bibfieldeditor
- Collins-Thompson Kevyn, 2014, TREC 2013 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Collins-Thompson Kevyn, 2015, TREC 2014 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Eickhoff Carsten, 2011, Proceedings of the workshop on crowdsourcing for search and data mining(CSDM) at the fourth ACM international conference on web search and data mining(WSDM). 11--14
- Hirth Matthias, 2010, Cheat-detection mechanisms for crowdsourcing
- Roberts Kirk, 2016, Overview of the TREC 2016 Clinical Decision Support Track. TREC.
- Tversky Amos, Judgment under uncertainty: Heuristics and biases. Utility, probability, and human decision making
- Vuurens Jeroen, 2011, ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR'11)
- Ye Peng, 2013, Conf. on Machine Learning. 1--7.
- Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20. Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20.
Cited by
Items 201–209 of 209
Lenart, R. (2026) · Edward Elgar Publishing eBooks · doi:10.4337/9781035388509.fm5
Data Annotation as Measurement
Harvey, E., Koenecke, A., & Kizilcec, R. F. (2026) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v9i2.43221
“…practices (Alab-duljabbar and Al-Dossari 2016; Alonso 2015b; Calderon, Reichart, and Dror 2025; Callison-Burch and Dredze 2010; Daniel et al. 2019; Eickhoff 2018; Hansen et al. 2013; Ho et al. 2015; Kovashka et al. 2016; Lease 2011; Li et al. 2017, 2019; Sabou et al. 2014; Snow et al. 2008;…”
Data for Machine Learning: A Human-Centered Perspective of Crowdsourcing
Law, E., & Yin, M. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.008
Doing Human-Centered Machine Learning
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.006
Advancing user satisfaction estimation in task-oriented dialogue system
Liang, H. (n.d.) · PolyU Institutional Research Archive (Hong Kong Polytechnic University)
Natural language processing for supporting impact assessment of funded projects
Han, K. (2025) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Los prebióticos y su efecto en el crecimiento de Lactobacillus: Una revisión actualizada
Torres Plasencia, P. E. (2026) · REBIOL
COTA: Confidence-aware Opinion and Trust Assessment
Tosi, M. D. L., Santis, A. D., Brambilla, M., & Cabot, J. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840617
Trapped by Expectations: Functional Fixedness in LLM-Enabled Chat Search
Liu, J., Karimnazarov, J., & White, R. W. (2025) · arXiv · doi:10.48550/arxiv.2504.02074
“Differing from previous works that have primarily examined cognitive biases in static search interactions [e.g., 30, 88], our work systematically investigates functional fixedness in dynamic, multi-turn chat search environments.”
Metadata
- Authors
- Eickhoff, C.
- Affiliations
- ETH Zurich (CH)
- Corresponding author
- Eickhoff, Carsten
- ORCID
- Eickhoff, Carsten
- Venue
- Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2018-02-02
- Pages
- 162-170
- OpenAlex
- W2783668259
- Cited by
- 144
- References
- 58
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 21 · 2023: 24 · 2022: 24 · 2021: 15 · 2020: 16 · 2019: 20 · 2018: 8
- FWCI
- 16.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- crowdsourcing, cognitive biases, quality control, annotation, data curation, information retrieval evaluation, cheating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.