Common Concerns with MTurk as a Participant Pool: Evidence and Solutions
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
In this chapter, we outline the common concerns with MTurk as a participant pool, review the evidence for those concerns, and discuss solutions. We close with a Table of considerations that researchers should make when fielding a study on MTurk
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our primary concern was the potential compromise in data quality and representativeness often associated with these platforms, as highlighted in recent studies [23,35,68].”
From Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models (2024) · cites this paper as [35]
Citations
Cites
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Self-citationInstructional manipulation checks: Detecting satisficing to increase statistical power
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2009.03.009 · Cited by 3,493
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172
Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2017.01.006 · Cited by 2,957
Determinants of social desirability bias in sensitive surveys: a literature review
Krumpal, I. (2013) · Quality & Quantity · doi:10.1007/s11135-011-9640-9 · Cited by 2,860
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips
Sparrow, B., Liu, J., & Wegner, D. M. (2011) · Science · doi:10.1126/science.1207745 · Cited by 1,762
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Self-citation
Cited by
Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01999-x · Cited by 269
Self-citationDevelopment and Initial Validation of the Duke Misophonia Questionnaire
Rosenthal, M. Z., Anand, D., Cassiello-Robbins, C., Williams, Z. J., Guetta, R. E., Trumbull, J., & Kelley, L. D. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.709928 · Cited by 117
Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Dhillon, P. S., Molaei, S., Li, J., Golub, M., Zheng, S., & Robert, L. P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642134 · Cited by 111
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
Lu, L., Neale, N., Line, N. D., & Bonn, M. (2022) · Cornell Hospitality Quarterly · doi:10.1177/19389655211025475 · Cited by 94
“Common criticisms include the possibility for skewed demographics (e.g., Berinsky et al., 2012; Paolacci et al., 2010), high rates of inattention (e.g., Goodman et al., 2013; Harms & DeSimone, 2015), and uncertain data quality (e.g., Hauser et al., 2018; Rouse, 2015), among others.”
BackgroundTrusting AI: does uncertainty visualization affect decision-making?
Reyes, J., Batmaz, A. U., & Kersten-Oertel, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1464348 · Cited by 27
Metadata
- Authors
- Hauser, D., Paolacci, G., & Chandler, J. J.
- Publisher
- Center for Open Science
- Published
- 2018-09-01
- OpenAlex
- W4234844921
- Cited by
- 124
- References
- 21
- Type
- posted content
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Common Concerns with MTurk as a Participant Pool: Evidence and Solutions. https://app.cochat.ai/papers/10.31234/osf.io/uq45c
- The paper
- Hauser, D., Paolacci, G., & Chandler, J. J. (2018). Common Concerns with MTurk as a Participant Pool: Evidence and Solutions. https://doi.org/10.31234/osf.io/uq45c
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.