The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Are samples drawn from Mechanical Turk valid for research on political ideology?
Open accessAbstract
Amazon’s Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the national public, we know little about whether liberals and conservatives recruited from MTurk share the same psychological dispositions as their counterparts in the mass public. In the absence of such evidence, some have argued that the selection process involved in joining MTurk invalidates the subject pool for studying questions central to political science. In this paper, we evaluate this claim by comparing a large MTurk sample to two benchmark national samples – one conducted online and one conducted face-to-face. We examine the personality and value-based motivations of political ideology across the three samples. All three samples produce substantively identical results with only minor variation in effect sizes. In short, liberals and conservatives in our MTurk sample closely mirror the psychological divisions of liberals and conservatives in the mass public, though MTurk liberals hold more characteristically liberal values and attitudes than liberals from representative samples. Overall, our results suggest that MTurk is a valid recruitment tool for psychological research on political ideology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The control of the false discovery rate in multiple testing under dependency
Benjamini, Y., & Yekutieli, D. (2001) · The Annals of Statistics · doi:10.1214/aos/1013699998 · Cited by 10,490
A very brief measure of the Big-Five personality domains
Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003) · Journal of Research in Personality · doi:10.1016/s0092-6566(03)00046-1 · Cited by 8,307
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242
Hauser, D. J., & Schwarz, N. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0578-z · Cited by 1,739
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
How Elite Partisan Polarization Affects Public Opinion Formation
DRUCKMAN, J. N., PETERSON, E., & SLOTHUUS, R. (2013) · American Political Science Review · doi:10.1017/s0003055412000500 · Cited by 1,569
Elite Influence on Public Opinion in an Informed Electorate
BULLOCK, J. G. (2011) · American Political Science Review · doi:10.1017/s0003055411000165 · Cited by 777 · cited 1×
“This would be particularly worrisome for research using ideology or partisanship as a moderator of experimental treatment effects (e.g., Bullock, 2011; Druckman et al., 2013; Nyhan and Reifler, 2010) or examining psychological differences between liberals and conservatives (e.g., Feldman and…”
Cited by
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Boas, T. C., Christenson, D. P., & Glick, D. M. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2018.28 · Cited by 570
The shape of and solutions to the MTurk quality crisis
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. G. (2020) · Political Science Research and Methods · doi:10.1017/psrm.2020.6 · Cited by 506
Self-citationAmericans, Not Partisans: Can Priming American National Identity Reduce Affective Polarization?
Levendusky, M. S. (2018) · The Journal of Politics · doi:10.1086/693987 · Cited by 492
Pennycook, G., & Rand, D. G. (2022) · Nature Communications · doi:10.1038/s41467-022-30073-5 · Cited by 270
Azevedo, F., Jost, J. T., Rothmund, T., & Sterling, J. (2019) · Journal of Social Issues · doi:10.1111/josi.12310 · Cited by 207
Metaphors for the War (or Race) against Climate Change
Flusberg, S. J., Matlock, T., & Thibodeau, P. H. (2017) · Environmental Communication · doi:10.1080/17524032.2017.1289111 · Cited by 203
Coping during the COVID-19 pandemic: Relations with mental health and quality of life.
Shamblaw, A. L., Rumas, R. L., & Best, M. W. (2021) · Canadian Psychology / Psychologie canadienne · doi:10.1037/cap0000263 · Cited by 156
Assessing public value failure in government adoption of artificial intelligence
Schiff, D. S., Schiff, K. J., & Pierson, P. (2022) · Public Administration · doi:10.1111/padm.12742 · Cited by 155
Bayesian or biased? Analytic thinking and political belief updating
Tappin, B. M., Pennycook, G., & Rand, D. G. (2020) · Cognition · doi:10.1016/j.cognition.2020.104375 · Cited by 127
Reducing affective polarization: Warm group relations or policy compromise?
Huddy, L., & Yair, O. (2021) · Political Psychology · doi:10.1111/pops.12699 · Cited by 124
Internet surveillance, regulation, and chilling effects online: a comparative case study
Penney, J. W. (2017) · Internet Policy Review · doi:10.14763/2017.2.692 · Cited by 122
Bouwman, R., & Grimmelikhuijsen, S. (2016) · International Journal of Public Sector Management · doi:10.1108/ijpsm-07-2015-0129 · Cited by 95
Pollet, T. V., & Saxton, T. K. (2019) · Evolutionary Psychological Science · doi:10.1007/s40806-019-00192-2 · Cited by 89
Moral polarization and out-party hostility in the US political context
Tappin, B. M., & McKay, R. T. (2019) · Journal of Social and Political Psychology · doi:10.5964/jspp.v7i1.1090 · Cited by 79
The social‐psychological bases of far‐right support in Europe and the United States
Van Assche, J., Dhont, K., & Pettigrew, T. F. (2019) · Journal of Community & Applied Social Psychology · doi:10.1002/casp.2407 · Cited by 69
Bowes, S. M., Blanchard, M. C., Costello, T. H., Abramowitz, A. I., & Lilienfeld, S. O. (2020) · Journal of Research in Personality · doi:10.1016/j.jrp.2020.103992 · Cited by 68
Representativeness and Response Validity Across Nine Opt-In Online Samples
Stagnaro, M. N., Druckman, J., Berinsky, A. J., Arechar, A. A., Willer, R., & Rand, D. G. (2024) · doi:10.31234/osf.io/h9j2d · Cited by 68
Experimentally induced social threat increases paranoid thinking
Saalfeld, V., Ramadan, Z., Bell, V., & Raihani, N. J. (2018) · Royal Society Open Science · doi:10.1098/rsos.180569 · Cited by 52
Limits to deployment of nuclear power for decarbonization: Insights from public opinion
Abdulla, A., Vaishnav, P., Sergi, B., & Victor, D. G. (2019) · Energy Policy · doi:10.1016/j.enpol.2019.03.039 · Cited by 50
Wong-Parodi, G. (2020) · Energy Research & Social Science · doi:10.1016/j.erss.2020.101757 · Cited by 48
Becker, B. (2020) · Social Justice Research · doi:10.1007/s11211-019-00343-7 · Cited by 37
Deep Alignment with Country Shrinks the Moral Gap Between Conservatives and Liberals
Talaifar, S., & Swann, W. B. (2019) · Political Psychology · doi:10.1111/pops.12534 · Cited by 23
The Role of Personality in Trust in Public Policy Automation
Waggoner, P., & Kennedy, R. (2022) · Journal of Behavioral Data Science · doi:10.35566/v2n1/p4/ · Cited by 7
Self-citationHow Partisan Crowds Affect News Evaluation
Jakesch, M., Koren, M., Evtushenko, A., & Naaman, M. (2022) · doi:10.31234/osf.io/baxts
Metadata
- Authors
- Clifford, S., Jewell, R. M., & Waggoner, P. D.
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2015-10-01
- Volume
- 2
- Issue
- 4
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W2196964148
- Cited by
- 686
- References
- 11
- FWCI
- 67.35
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
