Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 205

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool

Open access
Robinson, Jonathan, Rosenzweig, Cheskie, Moss, Aaron J., Litman, Leib·2019·PLOS ONE

Published version of a preprint

Abstract

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0226394

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 27

  1. M Buhrmester, 2011, Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data?, Perspect Psychol Sci, 6, 3, 10.1177/1745691610393980 · DOI · Paper page
  2. Bentley JW. Challenges with Amazon Mechanical Turk research in accounting. SSRN 2924876. 2018 Mar 30. · DOI
  3. JK Goodman, 2017, Crowdsourcing consumer research, J Consum Res, 44, 196, 10.1093/jcr/ucx047 · DOI · Paper page
  4. J Bohannon, 2016, Mechanical Turk upends social sciences, Science, 352, 1263, 10.1126/science.352.6291.1263 · DOI
  5. N Stewart, 2017, Crowdsourcing samples in cognitive science, Trends in Cogn Sci, 21, 736, 10.1016/j.tics.2017.06.007 · DOI · Paper page
  6. H Zhou, 2016, The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions, J Pers Soc Psychol, 111, 493, 10.1037/pspa0000056 · DOI · Paper page
  7. KA DeSoto, 2016, Under the hood of Mechanical Turk, APS Obs, 29
  8. Sorokin A, Forsyth D. Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2008 Jun 23 (pp. 1–8). IEEE. · DOI · Paper page
  9. J Bohannon, 2011, Social science for pennies, Science, 334, 307, 10.1126/science.334.6054.307 · DOI
  10. D Hauser, Handbook of research methods in consumer psychology
  11. J Chandler, 2019, Online panels in social science research: Expanding sampling methods beyond Mechanical Turk, Behav Res Methods, 11, 1
  12. Stagnaro M, Pennycook G, Rand DG. Performance on the Cognitive Reflection Test is stable across time. SSRN 3115809. 2018. · DOI
  13. KJ Mullinix, 2015, The generalizability of survey experiments, Journal of Experimental Political Science, 2, 109, 10.1017/XPS.2015.19 · DOI · Paper page
  14. DN Shapiro, 2013, Using Mechanical Turk to study clinical populations, Clin Psychol Sci, 1, 213, 10.1177/2167702612469015 · DOI · Paper page
  15. Difallah D, Filatova E, Ipeirotis P. Demographics and dynamics of Mechanical Turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining 2018 Feb 2 (pp. 135–143). ACM. · DOI · Paper page
  16. L Litman, 2017, TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences, Behavior Res Methods, 49, 433, 10.3758/s13428-016-0727-z · DOI · Paper page
  17. K Fort, 2011, Amazon Mechanical Turk: Gold mine or coal mine?, Comput Linguist, 37, 413, 10.1162/COLI_a_00057 · DOI · Paper page
  18. MA Smith, 1997, Virtual subjects: Using the Internet as an alternative source of subjects and research environment, Behav Res Methods Instrum Comput, 29, 496, 10.3758/BF03210601 · DOI · Paper page
  19. SR Ramsey, 2016, Psychological research in the internet age: The quality of web-based data, Comput Human Behav, 58, 354, 10.1016/j.chb.2015.12.049 · DOI · Paper page
  20. KE Jacowitz, 1995, Measures of anchoring in estimation tasks, Pers Soc Psychol Rev, 21, 1161, 10.1177/01461672952111004 · DOI · Paper page
  21. JJ Thomson, 1976, Killing, letting die, and the trolley problem, The Monist, 59, 204, 10.5840/monist197659224 · DOI · Paper page
  22. OP John, 2008, Handbook of personality: Theory and research, 114
  23. L Litman, 2015, The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk, Behavior Res Methods, 47, 519, 10.3758/s13428-014-0483-x · DOI · Paper page
  24. S Frederick, 2005, Cognitive reflection and decision making, J Econ Perspect, 19, 25, 10.1257/089533005775196732 · DOI · Paper page
  25. American National Election Studies, Stanford University, and University of Michigan, 2016, American National Election Study: 2016 Pilot Study
  26. Chandler J., L. Litman & Y. Robinson. Predicting retention in longitudinal studies conducted on Mechanical Turk." Presentation at the American Association for Public Opinion Research Conference, Toronto, May, 2019.
  27. Robinson J, Litman L, 2020. Online research on Mechanical Turk and other platforms. In Press. SAGE Publications.

Cited by

See all 205

Metadata

Authors
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L.
Venue
PLOS ONE
Published
2019-12-16
Volume
14
Issue
12
Pages
e0226394
ISSN
1932-6203 (ISSN-L)
OpenAlex
W2996509273
Cited by
205
References
12
FWCI
32.68
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. https://app.cochat.ai/papers/10.1371/journal.pone.0226394
The paper
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE. https://doi.org/10.1371/journal.pone.0226394

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.