Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Crowdsourced data acquired from tasks that comprise a subjective component (e.g. opinion detection, sentiment analysis) is potentially affected by the inherent bias of crowd workers who contribute to the tasks. This can lead to biased and noisy ground-truth data, propagating the undesirable bias and noise when used in turn to train machine learning models or evaluate systems. In this work, we aim to understand the influence of workers' own opinions on their performance in the subjective task of bias detection. We analyze the influence of workers' opinions on their annotations corresponding to different topics. Our findings reveal that workers with strong opinions tend to produce biased annotations. We show that such bias can be mitigated to improve the overall quality of the data collected. Experienced crowd workers also fail to distance themselves from their own opinions to provide unbiased annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [53]
Citations
Cites
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“To control for Type-I error inflation in our multiple comparisons, we used the Holm-Bonferroni correction for family-wise error rate (FWER) [21], at the significance Level of α < .”
MethodQualitative Analysis for Social Scientists
Strauss, A. L. (1987) · doi:10.1017/cbo9780511557842 · Cited by 12,277 · cited 1×
“We followed the process of open coding to ensure that the statements were reliably identified as pro, contra and neutral [50].”
MethodFraming as a Theory of Media Effects
Scheufele, D. A. (1999) · Journal of Communication · doi:10.1111/j.1460-2466.1999.tb02784.x · Cited by 2,179 · cited 1×
“For many tasks such as detecting subjective statements in text (i.e., text pieces reflecting opinions), or biased and framing issues that are often encountered in political discourse [13, 47], the quality of the ground-truth is crucial.”
BackgroundLanguage in the news: discourse and ideology in the press
(1991) · Choice Reviews Online · doi:10.5860/choice.29-1923 · Cited by 2,082
THE PRONOUNS OF POWER AND SOLIDARITY
Brown, R., & Gilman, A. (1968) · Readings in the Sociology of Language · doi:10.1515/9783110805376.252 · Cited by 1,782
Balance as bias: global warming and the US prestige press
Boykoff, M. T., & Boykoff, J. M. (2004) · Global Environmental Change · doi:10.1016/j.gloenvcha.2003.10.001 · Cited by 1,699 · cited 1×
“Studies [3, 4] show that the political or ideological stance of a person can influence the perception and interpretation of facts.”
BackgroundKittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 2×
“Microtask crowdsourcing provides remarkable opportunities to acquire human input at scale for a variety of purposes [35] including the creation of ground-truth data and the evaluation of systems.”
News, the politics of illusion
Bennett, W. L. (1983) · Cited by 1,024
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973 · cited 1×
“In large batches that consist of several similar tasks, Ipeirotis et al. showed that it is possible to use statistical methods and eliminate systematic bias [28].”
MethodSemin, G. R., & Fiedler, K. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.4.558 · Cited by 854 · cited 1×
“Subjective language [53] can be seen as a subproblem of language bias (e.g. framing, opinions etc.), which often is presented through subtle linguistic cues that carry an implicit sentiment [20, 48] and often are deliberately used in order to convey a specific stance towards a subject.”
BackgroundComputing Krippendorff's Alpha-Reliability
Krippendorff, K. (2011) · ScholarlyCommons (University of Pennsylvania) · Cited by 809
Wiebe, J., Wilson, T., Bruce, R., Bell, M., & Martin, M. (2004) · Computational Linguistics · doi:10.1162/0891201041850885 · Cited by 748 · cited 1×
“Subjective language [53] can be seen as a subproblem of language bias (e.g. framing, opinions etc.), which often is presented through subtle linguistic cues that carry an implicit sentiment [20, 48] and often are deliberately used in order to convey a specific stance towards a subject.”
BackgroundLanguage in Society: An Introduction to Sociolinguistics
Silva, D. J., & Romaine, S. (1994) · The Modern Language Journal · doi:10.2307/328616 · Cited by 593
More than fun and money. Worker motivation in crowdsourcing - a study on mechanical turk
Kaufmann, N., Schulze, T., & Veit, D. J. (2011) · MADOC (University of Mannheim) · Cited by 583
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · cited 2×
“Two popular methods to induce honest reporting in the absence of a ground-truth are the Bayesian truth serum method (BTS) [42] and the peerprediction method [40].”
MethodEliciting Informative Feedback: The Peer-Prediction Method
Miller, N., Resnick, P., & Zeckhauser, R. (2005) · Management Science · doi:10.1287/mnsc.1050.0379 · Cited by 506 · cited 1×
“Two popular methods to induce honest reporting in the absence of a ground-truth are the Bayesian truth serum method (BTS) [42] and the peer-prediction method [40].”
MethodIterative Learning for Reliable Crowdsourcing Systems
Karger, D. R., Oh, S., & Shah, D. (2011) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 468
Linguistic Models for Analyzing and Detecting Biased Language
Recasens, M. V., Danescu-Niculescu-Mizil, C., & Jurafsky, D. (2013) · Cited by 451
Baeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432 · cited 1×
“Recent work has reflected on the importance of creating an awareness of existing biases in order to alleviate the biases [2].”
BackgroundVariational Inference for Crowdsourcing
Liu, Q., Peng, J., & Ihler, A. (2012) · neural information processing systems · Cited by 376
Supervised learning from multiple experts: whom to trust when everyone lies a bit
Raykar, V. C., Yu, S., Zhao, L. H., Jerebko, A., Florin, C., Valadez, G. H., Bogoni, L., & Moy, L. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553488 · Cited by 333 · cited 1×
“Raykar et al. [43] introduce an approach for combining labels provided by multiple types of annotators (experts and novices) to obtain a final high quality label.”
BackgroundAn Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets
Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., & Vukovic, M. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v5i1.14105 · Cited by 299
Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Hossfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2291663 · Cited by 283 · cited 1×
“The generally low to moderate inter-worker agreement is consistent with expected agreement in similar tasks [24].”
ResultPolitical Ideology Detection Using Recursive Neural Networks
Iyyer, M., Enns, P., Boyd-Graber, J., & Resnik, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1105 · Cited by 275 · cited 1×
“In a similar study, Iyyer et al. [29] showed the impact of the workers in annotating statements with their corresponding political ideology.”
BackgroundAn Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets.
Rogstadius, J., Kostakos, V., Kittur, A., et al. (2011) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.13140/rg.2.2.19170.94401 · Cited by 268 · cited 1×
“Various factors ranging from worker motivation [34, 45] to marketplace dynamics such as task availability [8, 30], dictate which workers end up self-selecting and completing a given task from the available group of workers at any given point in time.”
BackgroundLanguage in Society: An Introduction to Sociolinguistics
Romaine, S. (2000) · doi:10.1093/oso/9780198731924.001.0001 · Cited by 249
Non-Conscious Influences on Consumer Choice
Fitzsimons, G. J., Hutchinson, J. W., Williams, P., Alba, J. W., Chartrand, T. L., Huber, J., Kardes, F. R., Menon, G., Raghubir, P., Russo, J. E., et al. (2002) · Marketing Letters · doi:10.1023/a:1020313710388 · Cited by 236
Designing incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236 · cited 1×
“In a related study, Shaw et al. found that when workers think about the responses that other workers give then they work more objectively [49].”
BackgroundNews: The Politics of Illusion, Tenth Edition
Bennett, W. L. (2016) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226345055.001.0001 · Cited by 226
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
Gadiraju, U., Kawase, R., Dietze, S., & Demartini, G. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702443 · Cited by 219 · cited 1×
“Although it is now common practice to deploy crowdsourcing jobs with quality control mechanisms embedded in them [18, 35], it is still uncommon to control for biases stemming from worker opinions.”
BackgroundSelf-citationHolmes, D. S. (1968) · Psychological Bulletin · doi:10.1037/h0025725 · Cited by 202 · cited 1×
“We draw inspiration from such truth-inducing methods as well as from the theory of social projection [22, 23] and aim to analyze the effect of social projection on mitigating biases stemming from worker opinions.”
MethodThe Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk
Difallah, D. E., Catasta, M., Demartini, G., Ipeirotis, P. G., & Cudré-Mauroux, P. (2015) · Proceedings of the 24th International Conference on World Wide Web · doi:10.1145/2736277.2741685 · Cited by 187 · cited 2×
“A survey of crowdsourcing tasks on Amazon’s Mechanical Turk [8] revealed that one of the most popular tasks is that of interpretation and analysis (IA) [17].”
BackgroundAn Evaluation of Aggregation Techniques in Crowdsourcing
Quoc Viet Hung, N., Tam, N. T., Tran, L. N., & Aberer, K. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-41154-0_1 · Cited by 177 · cited 1×
“One of the simplest methods used for aggregating multiple judgments in crowdsourced tasks is majority voting [27].”
MethodTask search in a human computation market
Chilton, L. B., Horton, J. J., Miller, R. C., & Azenkot, S. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837889 · Cited by 171 · cited 1×
“Consider a typical microtask crowdsourcing platform; task completion is generally driven by a self-selection process where workers pick and complete tasks they wish to [7].”
BackgroundIncreasing cheat robustness of crowdsourcing tasks
Eickhoff, C., & de Vries, A. P. (2013) · Information Retrieval · doi:10.1007/s10791-011-9181-9 · Cited by 167 · cited 1×
“This can be further exacerbated by incentive schemes [10].”
BackgroundMore than words: syntactic packaging and implicit sentiment
Greene, S., & Resnik, P. (2009) · Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics on - NAACL '09 · doi:10.3115/1620754.1620827 · Cited by 164
Cognitive Biases in Crowdsourcing
Eickhoff, C. (2018) · Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining · doi:10.1145/3159652.3159654 · Cited by 144 · cited 1×
“Eickhoff studied the prevalence of cognitive biases (ambiguity effect, anchoring, bandwagon and decoy effect) as a source of noise in crowdsourced data curation, annotation and evaluation [9].”
BackgroundBayesian Bias Mitigation for Crowdsourcing
Wauthier, F. L., & Jordan, M. I. (2011) · neural information processing systems · Cited by 137
A taxonomy of microtasks on the web
Gadiraju, U., Kawase, R., & Dietze, S. (2014) · Proceedings of the 25th ACM conference on Hypertext and social media · doi:10.1145/2631775.2631819 · Cited by 116 · cited 1×
“A survey of crowdsourcing tasks on Amazon’s Mechanical Turk [8] revealed that one of the most popular tasks is that of interpretation and analysis (IA) [17].”
BackgroundSelf-citationProjection as a defense mechanism.
Holmes, D. S. (1978) · Psychological Bulletin · doi:10.1037/0033-2909.85.4.677 · Cited by 111 · cited 1×
“We draw inspiration from such truth-inducing methods as well as from the theory of social projection [22, 23] and aim to analyze the effect of social projection on mitigating biases stemming from worker opinions.”
MethodDetecting Biased Statements in Wikipedia
Hube, C., & Fetahu, B. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3184558.3191640 · Cited by 94 · cited 3×
“We extracted biased statements from the English Wikipedia using the approach introduced by [44] and [25] for articles that cover the given topics, e.”
BackgroundMethodSelf-citationClarity is a Worthwhile Quality: On the Role of Task Clarity in Microtask Crowdsourcing
Gadiraju, U., Yang, J., & Bozzon, A. (2017) · Proceedings of the 28th ACM Conference on Hypertext and Social Media · doi:10.1145/3078714.3078715 · Cited by 87 · cited 1×
“Recent works have explored task related factors such as complexity and clarity that can influence and arguably bias the nature of task-related outcomes [19].”
BackgroundSelf-citationNeural Based Statement Classification for Biased Language
Hube, C., & Fetahu, B. (2019) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3289600.3291018 · Cited by 78 · cited 1×
“In our study of crowd worker bias we focus on the task of labeling biased statements, a task that has found prominence in recent times to create ground truth data and evaluate methods for bias detection in text ([25, 26, 44]).”
MethodSelf-citationExperiences surveying the crowd: reflections on methods, participation, and reliability
Marshall, C. C., & Shipman, F. M. (2013) · Proceedings of the 5th Annual ACM Web Science Conference · doi:10.1145/2464464.2464485 · Cited by 63 · cited 1×
“We also included two attention check questions to filter out inattentive workers [39].”
Identifying and Accounting for Task-Dependent Bias in Crowdsourcing
Kamar, E., Kapoor, A., & Horvitz, E. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13238 · Cited by 63 · cited 1×
“Kamar et al. introduced and evaluated probabilistic models for identifying and correcting task-dependent bias [32].”
BackgroundIncentives to Counter Bias in Human Computation
Faltings, B., Jurca, R., Pu, P., & Tran, B. D. (2014) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v2i1.13145 · Cited by 58 · cited 1×
“propose a game theoretic incentive scheme to counter the anchoring effect bias among workers [11].”
BackgroundJain, A., Sarma, A. D., Parameswaran, A., & Widom, J. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3067421.3067431 · Cited by 57 · cited 1×
“Various factors ranging from worker motivation [34, 45] to marketplace dynamics such as task availability [8, 30], dictate which workers end up self-selecting and completing a given task from the available group of workers at any given point in time.”
BackgroundUsing Worker Self-Assessments for Competence-Based Pre-Selection in Crowdsourcing Microtasks
Gadiraju, U., Fetahu, B., Kawase, R., Siehndel, P., & Dietze, S. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3119930 · Cited by 54 · cited 1×
“showed that some crowd workers exhibit inflated self-assessments due to a cognitive bias [16].”
BackgroundSelf-citationChain Reactions: The Impact of Order on Microtask Chains
Cai, C. J., Iqbal, S. T., & Teevan, J. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858237 · Cited by 51 · cited 1×
“Other works revealed a significant impact of task order on task outcomes [1, 6].”
BackgroundJiang, L., Wagner, C., & Nardi, B. A. (2015) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2015.98 · Cited by 50 · cited 1×
“Indian workers valued self-improvement benefits, whereas American workers valued emotional benefits [31].”
BackgroundModus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
Gadiraju, U., Checco, A., Gupta, N., & Demartini, G. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3130914 · Cited by 41 · cited 1×
“Work environments (i.e., the hardware and software affordances at the disposal of workers) have also shown to influence and bias task related outcomes such as completion time and work quality [15].”
BackgroundSelf-citationHow One Microtask Affects Another
Newell, E., & Ruths, D. A. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858490 · Cited by 34 · cited 1×
“Newell and Ruths showed that intertask effects could be a source of systematic bias in crowdsourced tasks [41].”
BackgroundLeveraging In-Batch Annotation Bias for Crowdsourced Active Learning
Zhuang, H., & Young, J. (2015) · Proceedings of the Eighth ACM International Conference on Web Search and Data Mining · doi:10.1145/2684822.2685301 · Cited by 30 · cited 1×
“Zhuang and Young [55] explore the impact of in-batch annotation bias, where items in a batch influence the labelling outcome of other items within the batch.”
BackgroundSimilarHITs: Revealing the Role of Task Similarity in Microtask Crowdsourcing
Aipe, A., & Gadiraju, U. (2018) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3209542.3209558 · Cited by 26 · cited 1×
“Other works revealed a significant impact of task order on task outcomes [1, 6].”
BackgroundSelf-citation
Research works printed but not linked: 9
- 10.4324/9781315002057 · doi:10.4324/9781315002057
- Liye Fu, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Tiebreaker: Using language models to quantify gender bias in sports journalism. arXiv preprint arXiv:1607.03895 (2016). · arXiv
- 10.5555/1620754.1620827 · doi:10.5555/1620754.1620827
- 10.5555/2986459.2986677 · doi:10.5555/2986459.2986677
- 10.5555/2999134.2999212 · doi:10.5555/2999134.2999212
- John Lyons. 1970. New horizons in linguistics, Volume. (1970).
- James Surowiecki. 2005. The wisdom of crowds. Anchor. · doi:10.5555/1095645
- 10.5555/2986459.2986660 · doi:10.5555/2986459.2986660
- 10.5555/1866696.1866719 · doi:10.5555/1866696.1866719
Cited by
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Self-citationData and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
The Data-Production Dispositif
Miceli, M., & Posada, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555561 · Cited by 118
“Some investigations have explored the role of worker subjectivity on datasets [7, 12, 29, 84] and have proposed ways to recognize and address worker bias [3, 28, 36, 84].”
BackgroundMirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
BackgroundMarkl, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533117 · Cited by 78
“Other research on crowd-sourcing also highlights the importance of balancing annotators from diverse backgrounds and accounting for variation in values and opinions [12, 69].”
Background
Metadata
- Authors
- Hube, C., Fetahu, B., & Gadiraju, U.
- Affiliations
- Leibniz University Hannover (DE)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-12
- OpenAlex
- W2941486823
- Cited by
- 129
- References
- 53
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 17 · 2023: 24 · 2022: 25 · 2021: 22 · 2020: 18 · 2019: 3
- FWCI
- 19.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- DESIR (731081)
- ERC Advanced Grant ALEXANDRIA (339233)
- AFEL (687916)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- crowdsourcing, subjective judgment, bias detection, annotation quality, bias mitigation, experienced workers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.