Cognitive Biases in Crowdsourcing
The DOI matches its Crossref record; 54 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Crowdsourcing has become a popular paradigm in data curation, annotation and evaluation for many artificial intelligence and information retrieval applications. Considerable efforts have gone into devising effective quality control mechanisms that identify or discourage cheat submissions in an attempt to improve the quality of noisy crowd judgments. Besides purposeful cheating, there is another source of noise that is often alluded to but insufficiently studied: Cognitive biases.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [36]
Citations
Cites
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Robertson, S. E., Walker, S., Jones, S., Hancock‐Beaulieu, M., & Gatford, M. (1994) · Text REtrieval Conference · Cited by 2,212
Adding Asymmetrically Dominated Alternatives: Violations of Regularity and the Similarity Hypothesis
Huber, J., Payne, J. W., & Puto, C. (1982) · Journal of Consumer Research · doi:10.1086/208899 · Cited by 1,943
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
A Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
Dawid, A. P., & Skene, A. M. (1979) · Applied Statistics · doi:10.2307/2346806 · Cited by 1,692 · cited 1×
“Following Dawid and Skene[12],whoinvestigatedisagreementsbetween diagnoses posed by multiple individual medical doctors, there have been several successful attempts at harnessing similar methods for crowdsourcing quality assurance [29, 51].”
Background(2008) · Cited by 1,657
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
TREC: Experiment and evaluation in information retrieval
Vicedo, J. L., & Gómez, J. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20583 · Cited by 837
A Deep Relevance Matching Model for Ad-hoc Retrieval
Guo, J., Fan, Y., Ai, Q., & Croft, W. B. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983769 · Cited by 806
Adapting boosting for information retrieval measures
Wu, Q., Burges, C. J. C., Svore, K. M., & Gao, J. (2010) · Information Retrieval · doi:10.1007/s10791-009-9112-1 · Cited by 567
Voorhees, E. M., & Harman, D. K. (2005) · The MIT Press eBooks · Cited by 392
Yan, Y., Rosales, R., Fung, G., & Dy, J. (2011) · Cited by 351
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334
Crowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
TREC: Experiment and Evaluation in Information Retrieval
(2007) · Online Information Review · doi:10.1108/14684520710832478 · Cited by 323
Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2014) · Operations Research · doi:10.1287/opre.2013.1235 · Cited by 297
Social Cognition: How Individuals Construct Social Reality
Greifeneder, R., Bless, H., & Fiedler, K. (2004) · Cited by 266
Worker types and personality traits in crowdsourcing relevance labels
Kazai, G., Kamps, J., & Milić-Frayling, N. (2011) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2063576.2063860 · Cited by 196
Quality through flow and immersion: gamifying crowdsourced relevance assessments
Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012) · Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2348283.2348400 · Cited by 190
Self-citationCrowdsourcing for book search evaluation: impact of hit design on comparative system ranking
Kazai, G., Kamps, J., Koolen, M., & Milić-Frayling, N. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2009947 · Cited by 184
Beliefs and biases in web search
White, R. (2013) · Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2484028.2484053 · Cited by 183
Crowdsourcing Document Relevance Assessment with Mechanical Turk
Grady, C. P. L., & Lease, M. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 175
On quality control and machine learning in crowdsourcing
Lease, M. (2011) · National Conference on Artificial Intelligence · Cited by 175
Pick-a-crowd: tell me what you like, and i'll tell you what to do
Difallah, D. E., Demartini, G., & Cudré-Mauroux, P. (2013) · Proceedings of the 22nd international conference on World Wide Web · doi:10.1145/2488388.2488421 · Cited by 173
Increasing cheat robustness of crowdsourcing tasks
Eickhoff, C., & de Vries, A. P. (2013) · Information Retrieval · doi:10.1007/s10791-011-9181-9 · Cited by 167
Self-citationHere or there: preference judgments for relevance
Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. (2008) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.5555/1793274.1793281 · Cited by 156
An analysis of human factors and label accuracy in crowdsourcing relevance judgments
Kazai, G., Kamps, J., & Milić-Frayling, N. (2012) · Information Retrieval · doi:10.1007/s10791-012-9205-0 · Cited by 155
Bayesian Bias Mitigation for Crowdsourcing
Wauthier, F. L., & Jordan, M. I. (2011) · neural information processing systems · Cited by 137
Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines
Sabou, M., Bontcheva, K., Derczynski, L., & Arno Scharl, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/569wpuc5pjnt · Cited by 128
Repeatable and reliable search system evaluation using crowdsourcing
Blanco, R., Halpin, H., Herzig, D. M., Mika, P., Pound, J., Thompson, H. S., & Duc, T. T. (2011) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2009916.2010039 · Cited by 111
Semi-Supervised Consensus Labeling for Crowdsourcing
Tang, W., & Lease, M. (2011) · Cited by 111
Overview of the TREC 2015 Clinical Decision Support Track.
Roberts, K. E., Simpson, M., Voorhees, E. M., & Hersh, W. (2015) · Text REtrieval Conference · Cited by 96
Carterette, B., Bennett, P. N., Chickering, D. M., & Dumais, S. (2008) · Springer eBooks · doi:10.1007/978-3-540-78646-7_5 · Cited by 81
On Aggregating Labels from Multiple Crowd Workers to Infer Relevance of Documents
Hosseini, M., Cox, I. J., Milić-Frayling, N., Kazai, G., & Vinay, V. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-28997-2_16 · Cited by 77
Harman, D. K. (1992) · ACM SIGIR Forum · doi:10.1145/146565.146567 · Cited by 77
Managing crowdsourced human computation: a tutorial
Ipeirotis, P. G., & Paritosh, P. K. (2011) · International World Wide Web Conference (WWW) · doi:10.1145/1963192.1963314 · Cited by 75
Cognitive Biases in Information Systems Research: A Scientometric Analysis
Fleischmann, M., Amirpur, M., Benlian, A., & Heß, T. (2014) · AIS Electronic Library (AISeL) · Cited by 71
How Crowdsourcable is Your Task
Eickhoff, C., & de Vries, A. P. (2011) · Data Archiving and Networked Services (DANS) · Cited by 67
Self-citationScaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement
Difallah, D., Catasta, M., Demartini, G., & Cudré-Mauroux, P. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13154 · Cited by 64
Incentives to Counter Bias in Human Computation
Faltings, B. V., Jurca, R., Pu, P., & Tran, B. D. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13145 · Cited by 57
Overview of the TREC 2014 Web Track
Collins‐Thompson, K., MacDonald, C. M., Bennett, P. N., Díaz, F., & Voorhees, E. M. (2017) · Text REtrieval Conference · Cited by 42
Estimating the Completion Time of Crowdsourced Tasks Using Survival Analysis Models
Wang, J., Faridani, S., & Ipeirotis, P. G. (2011) · Cited by 40
Leveraging In-Batch Annotation Bias for Crowdsourced Active Learning
Zhuang, H., & Young, J. (2015) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2684822.2685301 · Cited by 36
The ownership and reuse of visual media
Marshall, C. C., & Shipman, F. M., III (2011) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/1998076.1998108 · Cited by 35
Characterizing and detecting malicious crowdsourcing
Wang, T., Wang, G., Li, X., Zheng, H.-T., & Zhao, B. Y. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2491719 · Cited by 27
Labor allocation in paid crowdsourcing: experimental evidence on positioning, nudges and prices
Chandler, D. R., & Horton, J. J. (2011) · National Conference on Artificial Intelligence · Cited by 26
Crowdsourcing using Mechanical Turk: quality management and scalability
Ipeirotis, P. G. (2011) · doi:10.1145/1982624.1982625 · Cited by 25
Eickhoff, C. (2014) · doi:10.1145/2594776.2594788 · Cited by 22
Self-citationOn the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing
Kazai, G., & Milić-Frayling, N. (2009) · Cited by 19
Exploiting Document Content for Efficient Aggregation of Crowdsourcing Votes
Davtyan, M., Eickhoff, C., & Hofmann, T. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806460 · Cited by 18
Self-citationCharacterizing and detecting malicious crowdsourcing
Wang, T., Wang, G., Li, X., Zheng, H.-T., & Zhao, B. Y. (2013) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2534169.2491719 · Cited by 15
Active Content-Based Crowdsourcing Task Selection
Bansal, P., Eickhoff, C., & Hofmann, T. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983716 · Cited by 14
Self-citationRamanath, R., Choudhury, M., Bali, K., & Roy, R. S. (2013) · Cited by 9
ETH Zurich at TREC Clinical Decision Support 2016.
Greuter, S., Junker, P., Kuhn, L., Mance, F., Mermet, V., Rellstab, A., & Eickhoff, C. (2016) · Text REtrieval Conference · Cited by 4
Self-citation
Research works printed but not linked: 10
- Cyril Cleverdon, Readings in Information Retrieval. Readings in Information Retrieval, bibfieldeditor
- Collins-Thompson Kevyn, 2014, TREC 2013 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Collins-Thompson Kevyn, 2015, TREC 2014 web track overview. Technical Report. MICHIGAN UNIV ANN ARBOR.
- Eickhoff Carsten, 2011, Proceedings of the workshop on crowdsourcing for search and data mining(CSDM) at the fourth ACM international conference on web search and data mining(WSDM). 11--14
- Hirth Matthias, 2010, Cheat-detection mechanisms for crowdsourcing
- Roberts Kirk, 2016, Overview of the TREC 2016 Clinical Decision Support Track. TREC.
- Tversky Amos, Judgment under uncertainty: Heuristics and biases. Utility, probability, and human decision making
- Vuurens Jeroen, 2011, ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR'11)
- Ye Peng, 2013, Conf. on Machine Learning. 1--7.
- Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20. Dongqing Zhu and Ben Carterette . 2010. An analysis of assessor behavior in crowdsourced preference judgments SIGIR 2010 workshop on crowdsourcing for search evaluation. 17--20.
Cited by
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
BackgroundCulpepper, J. S., Diaz, F., & Smucker, M. D. (2018) · ACM SIGIR Forum · doi:10.1145/3274784.3274788 · Cited by 178
Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval
Azzopardi, L. (2021) · Proceedings of the 2021 Conference on Human Information Interaction and Retrieval · doi:10.1145/3406522.3446023 · Cited by 176
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Hube, C., Fetahu, B., & Gadiraju, U. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300637 · Cited by 129
“Eickhoff studied the prevalence of cognitive biases (ambiguity effect, anchoring, bandwagon and decoy effect) as a source of noise in crowdsourced data curation, annotation and evaluation [9].”
BackgroundStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Metadata
- Authors
- Eickhoff, C.
- Affiliations
- ETH Zurich (CH)
- Corresponding author
- Eickhoff, Carsten
- ORCID
- Eickhoff, Carsten
- Venue
- Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
- Publisher
- ACM
- Published
- 2018-02-02
- Pages
- 162-170
- OpenAlex
- W2783668259
- Cited by
- 144
- References
- 58
- Citations by year
- 2026: 6 · 2025: 10 · 2024: 21 · 2023: 24 · 2022: 24 · 2021: 15 · 2020: 16 · 2019: 20 · 2018: 8
- FWCI
- 16.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Web Search and Data Mining (WSDM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- crowdsourcing, cognitive biases, quality control, annotation, data curation, information retrieval evaluation, cheating
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.