The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Are samples drawn from Mechanical Turk valid for research on political ideology?
Open accessAbstract
Amazon’s Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the national public, we know little about whether liberals and conservatives recruited from MTurk share the same psychological dispositions as their counterparts in the mass public. In the absence of such evidence, some have argued that the selection process involved in joining MTurk invalidates the subject pool for studying questions central to political science. In this paper, we evaluate this claim by comparing a large MTurk sample to two benchmark national samples – one conducted online and one conducted face-to-face. We examine the personality and value-based motivations of political ideology across the three samples. All three samples produce substantively identical results with only minor variation in effect sizes. In short, liberals and conservatives in our MTurk sample closely mirror the psychological divisions of liberals and conservatives in the mass public, though MTurk liberals hold more characteristically liberal values and attitudes than liberals from representative samples. Overall, our results suggest that MTurk is a valid recruitment tool for psychological research on political ideology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
How Elite Partisan Polarization Affects Public Opinion Formation
DRUCKMAN, J. N., PETERSON, E., & SLOTHUUS, R. (2013) · American Political Science Review · doi:10.1017/s0003055412000500 · Cited by 1,569
Elite Influence on Public Opinion in an Informed Electorate
BULLOCK, J. G. (2011) · American Political Science Review · doi:10.1017/s0003055411000165 · Cited by 777 · cited 1×
“This would be particularly worrisome for research using ideology or partisanship as a moderator of experimental treatment effects (e.g., Bullock, 2011; Druckman et al., 2013; Nyhan and Reifler, 2010) or examining psychological differences between liberals and conservatives (e.g., Feldman and…”
Cited by
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Boas, T. C., Christenson, D. P., & Glick, D. M. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2018.28 · Cited by 570
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Self-citationAmericans, Not Partisans: Can Priming American National Identity Reduce Affective Polarization?
Levendusky, M. S. (2018) · The Journal of Politics · doi:10.1086/693987 · Cited by 492
Metadata
- Authors
- Clifford, S., Jewell, R. M., & Waggoner, P. D.
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2015-10-01
- Volume
- 2
- Issue
- 4
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W2196964148
- Cited by
- 686
- References
- 11
- FWCI
- 67.35
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Are samples drawn from Mechanical Turk valid for research on political ideology? https://app.cochat.ai/papers/10.1177/2053168015622072
- The paper
- Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015). Are samples drawn from Mechanical Turk valid for research on political ideology? Research & Politics. https://doi.org/10.1177/2053168015622072
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
