The DOI matches its Crossref record; 54 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Cognitive Biases in Crowdsourcing
Abstract
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [36]
Citations
Cites
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis
Huber, J., Payne, J. W., & Puto, C. (1982) · Journal of Consumer Research · doi:10.1086/208899 · Cited by 1,943
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Following Dawid and Skene[12],whoinvestigatedisagreementsbetween diagnoses posed by multiple individual medical doctors, there have been several successful attempts at harnessing similar methods for crowdsourcing quality assurance [29, 51].”
Background(2008) · Cited by 1,657
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806
Research works printed but not linked: 10
- Cyril Cleverdon, Readings in Information Retrieval. Readings in Information Retrieval, bibfieldeditor
- Collins-Thompson Kevyn, 2014, TREC 2013 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Collins-Thompson Kevyn, 2015, TREC 2014 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Eickhoff Carsten, 2011, Proceedings of the workshop on crowdsourcing for search and data mining(CSDM) at the fourth ACM international conference on web search and data mining(WSDM). 11--14
- Hirth Matthias, 2010, Cheat-detection mechanisms for crowdsourcing
- Roberts Kirk, 2016, Overview of the TREC 2016 Clinical Decision Support Track. TREC.
- Tversky Amos, Judgment under uncertainty: Heuristics and biases. Utility, probability, and human decision making
- Vuurens Jeroen, 2011, ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR'11)
- Ye Peng, 2013, Conf. on Machine Learning. 1--7.
- Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20. Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20.
Cited by
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
BackgroundCulpepper, J. S., Diaz, F., & Smucker, M. D. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274788 · Cited by 178
Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval
Azzopardi, L. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446023 · Cited by 176
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129
“Eickhoff studied the prevalence of cognitive biases (ambiguity effect, anchoring, bandwagon and decoy effect) as a source of noise in crowdsourced data curation, annotation and evaluation [9].”
BackgroundStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Metadata
- Authors
- Eickhoff, C.
- Affiliations
- ETH Zurich (CH)
- Corresponding author
- Eickhoff, Carsten
- ORCID
- Eickhoff, Carsten
- Venue
- Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2018-02-02
- Pages
- 162-170
- OpenAlex
- W2783668259
- Cited by
- 144
- References
- 58
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 21 · 2023: 24 · 2022: 24 · 2021: 15 · 2020: 16 · 2019: 20 · 2018: 8
- FWCI
- 16.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- crowdsourcing, cognitive biases, quality control, annotation, data curation, information retrieval evaluation, cheating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Cognitive Biases in Crowdsourcing. https://app.cochat.ai/papers/10.1145/3159652.3159654
- The paper
- Eickhoff, C. (2018). Cognitive Biases in Crowdsourcing. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. https://doi.org/10.1145/3159652.3159654
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.