The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk
Abstract
In the past half-decade, Amazon Mechanical Turk has radically changed the way many scholars do research. The availability of a massive, distributed, anonymous crowd of individuals willing to perform general human-intelligence micro-tasks for micro-payments is a valuable resource for researchers and practitioners. This paper addresses the challenges of obtaining quality annotations for subjective judgment oriented tasks of varying difficulty. We design and conduct a large, controlled experiment (N=68,000) to measure the efficacy of selected strategies for obtaining high quality data annotations from non-experts. Our results point to the advantages of person-oriented strategies over process-oriented strategies. Specifically, we find that screening workers for requisite cognitive aptitudes and providing training in qualitative coding techniques is quite effective, significantly outperforming control and baseline conditions. Interestingly, such strategies can improve coder annotation accuracy above and beyond common benchmark strategies such as Bayesian Truth Serum (BTS).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The growing use of ML across industries, and the high-stakes nature of some of the above uses [17,44], is being accompanied by greater scrutiny of the processes that determine the output of ML-based decision-support systems [51].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [51]
Citations
Cites
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
The Coding Manual for Qualitative Researchers
Saldana, J. (2025) · SAGE Publications Ltd eBooks · doi:10.4135/9781036235611 · Cited by 17,944
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Self-citationEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Information credibility on twitter
Castillo, C., Mendoza, M., & Poblete, B. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963500 · Cited by 2,593
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Research works printed but not linked: 6
- Lau J.H., 2012, COLING
- Qazvinian V., EMNLP2011
- Rashtchian C., 2010, Proc. NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 139--147
- 10.5555/1613715.1613751 · doi:10.5555/1613715.1613751
- Sun Y.-A., 2011, Human Computation
- Surowiecki J., 2004, The Wisdom of Crowds
Cited by
Empath: Understanding Topic Signals in Large-Scale Text
Fast, E., Chen, B., & Bernstein, M. S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858535 · Cited by 413
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models
Zhang, H., Song, H., Li, S., Zhou, M., & Song, D. (2024) · ACM Computing Surveys · doi:10.1145/3617680 · Cited by 317
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
Metadata
- Authors
- Mitra, T., Hutto, C. J., & Gilbert, E.
- Affiliations
- Georgia Institute of Technology (US)
- Georgia Institute of Technology, Atlanta, GA, USA
- ORCID
- Mitra, Tanushree
- Venue
- Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2015-04-18
- Pages
- 1345-1354
- OpenAlex
- W2110151287
- Semantic Scholar
- 79478d8706d7b6ed5f5d858b51461f676bc9e315
- Cited by
- 102
- References
- 45
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 3 · 2023: 12 · 2022: 3 · 2021: 20 · 2020: 13 · 2019: 9 · 2018: 11 · 2017: 12 · 2016: 12 · 2015: 1
- FWCI
- 8.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- DARPA (W911NF-12-1-0043)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Amazon Mechanical Turk, crowdsourcing, annotation quality, human intelligence tasks, worker screening, cognitive ability, coder training, qualitative coding, Bayesian Truth Serum, subjective judgment, non-expert annotators
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. https://app.cochat.ai/papers/10.1145/2702123.2702553
- The paper
- Mitra, T., Hutto, C. J., & Gilbert, E. (2015). Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2702123.2702553
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.