The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
Abstract
Crowdsourced data acquired from tasks that comprise a subjective component (e.g. opinion detection, sentiment analysis) is potentially affected by the inherent bias of crowd workers who contribute to the tasks. This can lead to biased and noisy ground-truth data, propagating the undesirable bias and noise when used in turn to train machine learning models or evaluate systems. In this work, we aim to understand the influence of workers' own opinions on their performance in the subjective task of bias detection. We analyze the influence of workers' opinions on their annotations corresponding to different topics. Our findings reveal that workers with strong opinions tend to produce biased annotations. We show that such bias can be mitigated to improve the overall quality of the data collected. Experienced crowd workers also fail to distance themselves from their own opinions to provide unbiased annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“it has been shown that crowdworkers' personal opinions, demographic characteristics and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as [53]
Citations
Cites
A Simple Sequentially Rejective Multiple Test Procedure
(1979) · Scandinavian Journal of Statistics · doi:10.2307/4615733 · Cited by 21,526 · cited 1×
“To control for Type-I error inflation in our multiple comparisons, we used the Holm-Bonferroni correction for family-wise error rate (FWER) [21], at the significance Level of α < .”
MethodQualitative Analysis for Social Scientists
Strauss, A. L. (1987) · doi:10.1017/cbo9780511557842 · Cited by 12,277 · cited 1×
“We followed the process of open coding to ensure that the statements were reliably identified as pro, contra and neutral [50].”
MethodFraming as a Theory of Media Effects
Scheufele, D. A. (1999) · Journal of Communication · doi:10.1111/j.1460-2466.1999.tb02784.x · Cited by 2,179 · cited 1×
“For many tasks such as detecting subjective statements in text (i.e., text pieces reflecting opinions), or biased and framing issues that are often encountered in political discourse [13, 47], the quality of the ground-truth is crucial.”
BackgroundLanguage in the news: discourse and ideology in the press
(1991) · Choice Reviews Online · doi:10.5860/choice.29-1923 · Cited by 2,082
THE PRONOUNS OF POWER AND SOLIDARITY
Brown, R., & Gilman, A. (1968) · Readings in the Sociology of Language · doi:10.1515/9783110805376.252 · Cited by 1,782
Balance as bias: global warming and the US prestige press
Boykoff, M. T., & Boykoff, J. M. (2004) · Global Environmental Change · doi:10.1016/j.gloenvcha.2003.10.001 · Cited by 1,699 · cited 1×
“Studies [3, 4] show that the political or ideological stance of a person can influence the perception and interpretation of facts.”
BackgroundKittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 2×
“Microtask crowdsourcing provides remarkable opportunities to acquire human input at scale for a variety of purposes [35] including the creation of ground-truth data and the evaluation of systems.”
News, the politics of illusion
Bennett, W. L. (1983) · Cited by 1,024
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973 · cited 1×
“In large batches that consist of several similar tasks, Ipeirotis et al. showed that it is possible to use statistical methods and eliminate systematic bias [28].”
MethodSemin, G. R., & Fiedler, K. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.4.558 · Cited by 854 · cited 1×
“Subjective language [53] can be seen as a subproblem of language bias (e.g. framing, opinions etc.), which often is presented through subtle linguistic cues that carry an implicit sentiment [20, 48] and often are deliberately used in order to convey a specific stance towards a subject.”
Background
Research works printed but not linked: 10
- 10.4324/9781315002057 · doi:10.4324/9781315002057
- Liye Fu, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Tiebreaker: Using language models to quantify gender bias in sports journalism. arXiv preprint arXiv:1607.03895 (2016). · arXiv
- 10.5555/1620754.1620827 · doi:10.5555/1620754.1620827
- 10.5555/2986459.2986677 · doi:10.5555/2986459.2986677
- 10.5555/2999134.2999212 · doi:10.5555/2999134.2999212
- John Lyons. 1970. New horizons in linguistics, Volume. (1970).
- Drazen Prelec. 2004. A Bayesian truth serum for subjective data. science 306, 5695 (2004), 462--466.
- James Surowiecki. 2005. The wisdom of crowds. Anchor. · doi:10.5555/1095645
- 10.5555/2986459.2986660 · doi:10.5555/2986459.2986660
- 10.5555/1866696.1866719 · doi:10.5555/1866696.1866719
Cited by
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043
Self-citationData and its (dis)contents: A survey of dataset development and use in machine learning research
Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100336 · Cited by 548
The Data-Production Dispositif
Miceli, M., & Posada, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555561 · Cited by 267
“Some investigations have explored the role of worker subjectivity on datasets [7, 12, 29, 84] and have proposed ways to recognize and address worker bias [3, 28, 36, 84].”
BackgroundMirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240
“and cognitive biases [36] can affect the quality of crowdsourced annotations in fact-checking [30] or in tasks involving subjective assessments [53].”
BackgroundMarkl, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533117 · Cited by 153
“Other research on crowd-sourcing also highlights the importance of balancing annotators from diverse backgrounds and accounting for variation in values and opinions [12, 69].”
Background
Metadata
- Authors
- Hube, C., Fetahu, B., & Gadiraju, U.
- Affiliations
- Leibniz University Hannover (DE)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-12
- OpenAlex
- W2941486823
- Cited by
- 129
- References
- 53
- Citations by year
- 2026: 7 · 2025: 13 · 2024: 17 · 2023: 24 · 2022: 25 · 2021: 22 · 2020: 18 · 2019: 3
- FWCI
- 19.68
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- DESIR (731081)
- ERC Advanced Grant ALEXANDRIA (339233)
- AFEL (687916)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
- Keywords
- crowdsourcing, subjective judgment, bias detection, annotation quality, bias mitigation, experienced workers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. https://app.cochat.ai/papers/10.1145/3290605.3300637
- The paper
- Hube, C., Fetahu, B., & Gadiraju, U. (2019). Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300637
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.