The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool
Open accessPublished version of a preprint
Abstract
Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Framing of Decisions and the Psychology of Choice
Tversky, A., & Kahneman, D. (1981) · Science · doi:10.1126/science.7455683 · Cited by 18,497 · cited 1×
“The Asian Disease problem is a classic framing effect [32].”
BackgroundEvaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · cited 6×
“Although we did not conduct a formal power analysis, we aimed to recruit large samples in line with past work examining data quality on MTurk [1, 28].”
BackgroundResultInfluentialEvaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
A Dissociation Between Moral Judgments and Justifications
HAUSER, M., CUSHMAN, F., YOUNG, L., KANG‐XING JIN, R., & MIKHAIL, J. (2007) · Mind & Language · doi:10.1111/j.1468-0017.2006.00297.x · Cited by 666
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
Coppock, A. (2019) · Political Science Research and Methods · doi:10.1017/psrm.2018.10 · Cited by 515
Research works printed but not linked: 27
- M Buhrmester, 2011, Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?, Perspect Psychol Sci, 6, 3, 10.1177/1745691610393980 · DOI · Paper page
- Bentley JW. Challenges with Amazon Mechanical Turk research in accounting. SSRN 2924876. 2018 Mar 30. · DOI
- JK Goodman, 2017, Crowdsourcing consumer research, J Consum Res, 44, 196, 10.1093/jcr/ucx047 · DOI · Paper page
- J Bohannon, 2016, Mechanical Turk upends social sciences, Science, 352, 1263, 10.1126/science.352.6291.1263 · DOI
- N Stewart, 2017, Crowdsourcing samples in cognitive science, Trends in Cogn Sci, 21, 736, 10.1016/j.tics.2017.06.007 · DOI · Paper page
- H Zhou, 2016, The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions, J Pers Soc Psychol, 111, 493, 10.1037/pspa0000056 · DOI · Paper page
- KA DeSoto, 2016, Under the hood of Mechanical Turk, APS Obs, 29
- Sorokin A, Forsyth D. Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2008 Jun 23 (pp. 1–8). IEEE. · DOI · Paper page
- J Bohannon, 2011, Social science for pennies, Science, 334, 307, 10.1126/science.334.6054.307 · DOI
- D Hauser, Handbook of research methods in consumer psychology
- J Chandler, 2019, Online panels in social science research: Expanding sampling methods beyond Mechanical Turk, Behav Res Methods, 11, 1
- Stagnaro M, Pennycook G, Rand DG. Performance on the Cognitive Reflection Test is stable across time. SSRN 3115809. 2018. · DOI
- KJ Mullinix, 2015, The generalizability of survey experiments, Journal of Experimental Political Science, 2, 109, 10.1017/XPS.2015.19 · DOI · Paper page
- DN Shapiro, 2013, Using Mechanical Turk to study clinical populations, Clin Psychol Sci, 1, 213, 10.1177/2167702612469015 · DOI · Paper page
- Difallah D, Filatova E, Ipeirotis P. Demographics and dynamics of Mechanical Turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining 2018 Feb 2 (pp. 135–143). ACM. · DOI · Paper page
- L Litman, 2017, TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences, Behavior Res Methods, 49, 433, 10.3758/s13428-016-0727-z · DOI · Paper page
- K Fort, 2011, Amazon Mechanical Turk: Gold mine or coal mine?, Comput Linguist, 37, 413, 10.1162/COLI_a_00057 · DOI · Paper page
- MA Smith, 1997, Virtual subjects: Using the Internet as an alternative source of subjects and research environment, Behav Res Methods Instrum Comput, 29, 496, 10.3758/BF03210601 · DOI · Paper page
- SR Ramsey, 2016, Psychological research in the internet age: The quality of web-based data, Comput Human Behav, 58, 354, 10.1016/j.chb.2015.12.049 · DOI · Paper page
- KE Jacowitz, 1995, Measures of anchoring in estimation tasks, Pers Soc Psychol Rev, 21, 1161, 10.1177/01461672952111004 · DOI · Paper page
- JJ Thomson, 1976, Killing, letting die, and the trolley problem, The Monist, 59, 204, 10.5840/monist197659224 · DOI · Paper page
- OP John, 2008, Handbook of personality: Theory and research, 114
- L Litman, 2015, The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk, Behavior Res Methods, 47, 519, 10.3758/s13428-014-0483-x · DOI · Paper page
- S Frederick, 2005, Cognitive reflection and decision making, J Econ Perspect, 19, 25, 10.1257/089533005775196732 · DOI · Paper page
- American National Election Studies, Stanford University, and University of Michigan, 2016, American National Election Study: 2016 Pilot Study
- Chandler J., L. Litman & Y. Robinson. Predicting retention in longitudinal studies conducted on Mechanical Turk." Presentation at the American Association for Public Opinion Research Conference, Toronto, May, 2019.
- Robinson J, Litman L, 2020. Online research on Mechanical Turk and other platforms. In Press. SAGE Publications.
Cited by
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Self-citationEvaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 269
Self-citationComparing attentional disengagement between prolific and MTurk samples
Albert, D. A., & Smilek, D. (2023) · Scientific Reports · doi:10.1038/s41598-023-46048-5 · Cited by 166
Elbogen, E. B., Lanier, M., Blakey, S. M., Wagner, H. R., & Tsai, J. (2021) · Depression and Anxiety · doi:10.1002/da.23162 · Cited by 142
Sorkin, D. H., Janio, E. A., Eikey, E. V., Schneider, M., Davis, K., Schueller, S. M., Stadnick, N. A., Zheng, K., Neary, M., Safani, D., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/26994 · Cited by 108
Metadata
- Authors
- Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-12-16
- Volume
- 14
- Issue
- 12
- Pages
- e0226394
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2996509273
- Cited by
- 205
- References
- 12
- FWCI
- 32.68
- Type
- journal article
- Language
- en
- Update notices
- correction (2026-02-04)
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. https://app.cochat.ai/papers/10.1371/journal.pone.0226394
- The paper
- Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE. https://doi.org/10.1371/journal.pone.0226394
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.