Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk
The DOI matches its Crossref record; 45 of the 47 references it cites were found in a registry or an index.
Abstract
In the past half-decade, Amazon Mechanical Turk has radically changed the way many scholars do research. The availability of a massive, distributed, anonymous crowd of individuals willing to perform general human-intelligence micro-tasks for micro-payments is a valuable resource for researchers and practitioners. This paper addresses the challenges of obtaining quality annotations for subjective judgment oriented tasks of varying difficulty. We design and conduct a large, controlled experiment (N=68,000) to measure the efficacy of selected strategies for obtaining high quality data annotations from non-experts. Our results point to the advantages of person-oriented strategies over process-oriented strategies. Specifically, we find that screening workers for requisite cognitive aptitudes and providing training in qualitative coding techniques is quite effective, significantly outperforming control and baseline conditions. Interestingly, such strategies can improve coder annotation accuracy above and beyond common benchmark strategies such as Bayesian Truth Serum (BTS).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The growing use of ML across industries, and the high-stakes nature of some of the above uses [17,44], is being accompanied by greater scrutiny of the processes that determine the output of ML-based decision-support systems [51].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [51]
Citations
Cites
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
The Coding Manual for Qualitative Researchers
Saldana, J. (2025) · SAGE Publications Ltd eBooks · doi:10.4135/9781036235611 · Cited by 17,944
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Self-citationEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 1×
“, experience with more than 100 HITs and 95% approval ratings) is a common practice among scholars using AMT [1,6,11,27].”
BackgroundConducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Information credibility on twitter
Castillo, C., Mendoza, M., & Poblete, B. (2011) · Proceedings of the 20th international conference on World wide web · doi:10.1145/1963405.1963500 · Cited by 2,593
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
(2009) · Cited by 1,900
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
A Trainable System for Object Detection
Papageorgiou, C., & Poggio, T. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008162616689 · Cited by 1,289
Comparing Twitter and Traditional Media Using Topic Models
Zhao, W. X., Jiang, J., Weng, J., He, J., Lim, E.-P., Yan, H., & Li, X. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-20161-5_34 · Cited by 1,214
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973
Developing a Test Collection for the Evaluation of Integrated Search
Lykke, M., Larsen, B., Lund, H., & Ingwersen, P. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_63 · Cited by 969
Analyzing the Amazon Mechanical Turk marketplace
Ipeirotis, P. G. (2010) · XRDS: Crossroads, The ACM Magazine for Students · doi:10.1145/1869086.1869094 · Cited by 879
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
Heer, J., & Bostock, M. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753357 · Cited by 816
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 789
(2011) · Cited by 724
Is it really about me?: message content in social awareness streams
Naaman, M., Boase, J., & Lai, C.-H. (2010) · Proceedings of the 2010 ACM conference on Computer supported cooperative work · doi:10.1145/1718918.1718953 · Cited by 720
Utility data annotation with Amazon Mechanical Turk
Sorokin, A., & Forsyth, D. (2008) · 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562953 · Cited by 616
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · cited 1×
“They were told that their probability of getting a bonus would be higher if they submit answers that are more surprisingly common (the same wording as [30,35]).”
BackgroundFinancial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809422 · Cited by 463
Type I Error Rate Comparisons of Post Hoc Procedures for I j Chi-Square Tables
MacDonald, P. L., & Gardner, R. C. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970871 · Cited by 456
Are your participants gaming the system?: screening mechanical turk workers
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 418
Tweeting is believing?: understanding microblog credibility perceptions
Morris, M. R., Counts, S., Roseway, A., Hoff, A., & Schwarz, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145274 · Cited by 411
Wang, Y.-C., Kraut, R., & Levine, J. M. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145329 · Cited by 282
Designing incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236
On-line Trend Analysis with Topic Models: #twitter Trends Detection Topic Model Online
Lau, J. H., Collier, N., & Baldwin, T. (2012) · International Conference on Computational Linguistics · Cited by 231
SQUARE: A Benchmark for Research on Computing Crowd Consensus
Sheshadri, A., & Lease, M. (2013) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v1i1.13088 · Cited by 177
Exploring iterative and parallel human computation processes
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837907 · Cited by 166
Worker types and personality traits in crowdsourcing relevance labels
Kazai, G., Kamps, J., & Milic-Frayling, N. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063860 · Cited by 153
Strategies for crowdsourcing social data analysis
Willett, W., Heer, J., & Agrawala, M. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207709 · Cited by 103
Glance: rapidly coding behavioral video with the crowd
Lasecki, W. S., Gordon, M., Koutra, D., Jung, M. F., Dow, S. P., & Bigham, J. P. (2014) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2642918.2647367 · Cited by 94
Multiple contrasts and ordered Bonferroni procedures.
Rosenthal, R., & Rubin, D. B. (1984) · Journal of Educational Psychology · doi:10.1037//0022-0663.76.6.1028 · Cited by 90
Huang, S., & Fu, W. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470743 · Cited by 78
Crowd synthesis: extracting categories and clusters from complex data
André, P., Kittur, A., & Dow, S. P. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531653 · Cited by 70
Integrating on-demand fact-checking with public dialogue
Kriplean, T., Bonnar, C., Borning, A., Kinney, B., & Gill, B. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531677 · Cited by 65
Modeling Factuality Judgments in Social Media Text
Soni, S., Mitra, T., Gilbert, É., & Eisenstein, J. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2068 · Cited by 61
Self-citationSun, Y., Roy, S., & Little, G. (2011) · Cited by 24
What if we ask a different question? social inferences create product ratings faster
Gilbert, É. (2014) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2556288.2557081 · Cited by 7
Research works printed but not linked: 6
- Lau J.H., 2012, COLING
- Qazvinian V., EMNLP2011
- Rashtchian C., 2010, Proc. NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 139--147
- 10.5555/1613715.1613751 · doi:10.5555/1613715.1613751
- Sun Y.-A., 2011, Human Computation
- Surowiecki J., 2004, The Wisdom of Crowds
Cited by
Empath: Understanding Topic Signals in Large-Scale Text
Fast, E., Chen, B., & Bernstein, M. S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858535 · Cited by 413
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models
Zhang, H., Song, H., Li, S., Zhou, M., & Song, D. (2024) · ACM Computing Surveys · doi:10.1145/3617680 · Cited by 317
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, E., & Loaiza-Ganem, G. (2023) · neural information processing systems · doi:10.52202/075280-0165 · Cited by 290
Metadata
- Authors
- Mitra, T., Hutto, C. J., & Gilbert, E.
- Affiliations
- Georgia Institute of Technology (US)
- Georgia Institute of Technology, Atlanta, GA, USA
- ORCID
- Mitra, Tanushree
- Venue
- Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2015-04-18
- Pages
- 1345-1354
- OpenAlex
- W2110151287
- Semantic Scholar
- 79478d8706d7b6ed5f5d858b51461f676bc9e315
- Cited by
- 102
- References
- 47
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 3 · 2023: 12 · 2022: 3 · 2021: 20 · 2020: 13 · 2019: 9 · 2018: 11 · 2017: 12 · 2016: 12 · 2015: 1
- FWCI
- 8.56
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- DARPA (W911NF-12-1-0043)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Amazon Mechanical Turk, crowdsourcing, annotation quality, human intelligence tasks, worker screening, cognitive ability, coder training, qualitative coding, Bayesian Truth Serum, subjective judgment, non-expert annotators
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.