The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
The Demographic and Political Composition of Mechanical Turk Samples
Open accessAbstract
One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Market research quota sampling is a non-probabilistic sampling strategy whereby participants are identified from a database to match the population of interest (a discussion of the use of these estimates is in the appendix [p 7]).10 Samples obtained via this method might produce results that differ from the general population.11, 12”
From Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic (2021) · cites this paper as [12]
In the citing paper
“MTurk -based samples have also been criticized as being younger, more educated, and less politically and religiously diverse than the general population (Levay et al., 2016), potentially limiting generalizability.”
From Cultural humility and cultural competence in counseling: An exploratory mixed methods investigation (2023) · cites this paper as Levay et al., 2016
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · cited 20×
“Following others (e.g., Berinsky et al., 2012; Huff & Tingley, 2015; Krupnikov & Levine, 2014), we set three requirements for participation: a HIT Approval Rate greater than or equal to 90, at least 50 HITs approved, and an IP address based in the United States.”
BackgroundMethodResultInfluentialRunning experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 3×
“…with 45% of the MTurk sample possessing at least a bachelor’s degree versus only 29.5% of the ANES sample.8 Interestingly, while some earlier work highlights that MTurk samples underrepresent men (e.g., Ipeirotis, 2010; Paolacci et al., 2010), our data exhibit the opposite pattern (53.9% male).”
BackgroundMethodResultInfluentialInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,078
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 2×
“This finding, however, is consistent with more recent work (e.g., Berinsky et al., 2012; Casler et al., 2013; Krupnikov & Levine, 2014).”
MethodResultThe Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228 · cited 6×
“…& Levine, 2014; Paolacci & Chandler, 2014).3 Similarly, differences in political identity appear to be mostly stable, with MTurk samples frequently containing many more Democrats and liberals than populationbased samples (Berinsky et al., 2012; Krupnikov & Levine, 2014; Mullinix et al., 2015).”
BackgroundMethodInfluentialSelf-citationHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · cited 11×
“With the above in mind, as highlighted by others (e.g., Huff & Tingley, 2015), a promising approach for researchers concerned with representativeness or with specific sample needs is to employ software and sampling procedures aimed at creating larger and/or targeted survey pools.”
BackgroundMethodResultInfluentialThe Unheavenly Chorus: Unequal Political Voice and the Broken Promise of American Democracy
Schlozman, K. L., Verba, S., & Brady, H. E. (2012) · Princeton University Press eBooks · doi:10.1515/9781400841912 · Cited by 701
Population-Based Survey Experiments
Mutz, D. C. (2011) · doi:10.1515/9781400840489 · Cited by 674
Research works printed but not linked: 2
- Leeper T. J., 2013, The Political Methodologist, 2, 2
- Mason W. A., 2009, Association for Computing Machinery Explorations Newsletter, 11, 100
Cited by
Santomauro, D. F., Mantilla Herrera, A. M., Shadid, J., Zheng, P., Ashbaugh, C., Pigott, D. M., Abbafati, C., Adolph, C., Amlag, J. O., Aravkin, A. Y., et al. (2021) · The Lancet · doi:10.1016/s0140-6736(21)02143-7 · Cited by 5,444
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
“…work, we need probability samples, but for research that models the relationships between variables, convenience samples may be acceptable.2 Levay et al. (2016) provides some empirical support for the compromise’s underlying reasoning: MTurk and probability samples are descriptively quite…”
BackgroundMTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
“…model-based poststratification can be used to statistically adjust the estimates obtained from nonrepresentative samples (Battaglia, Hoaglin, and Frankel 2013; Park, Gelman, and Bafumi 2004), and may be applied to MTurk samples (Goel, Obeng, and Rothschild 2015; Levay, Freese, and Druckman 2016).”
Background
Metadata
- Authors
- Levay, K. E., Freese, J., & Druckman, J. N.
- Affiliations
- Northwestern University (US)
- Stanford University, CA, USA
- Corresponding author
- Druckman, James N.
- ORCID
- Druckman, James N.
- Venue
- Sage Open
- Publisher
- SAGE Publications
- Published
- 2016-01-01
- Volume
- 6
- Issue
- 1
- ISSN
- 2158-2440 (ISSN-L)
- OpenAlex
- W2301926123
- Cited by
- 676
- References
- 26
- Citations by year
- 2026: 21 · 2025: 48 · 2024: 58 · 2023: 75 · 2022: 88 · 2021: 103 · 2020: 103 · 2019: 72 · 2018: 61 · 2017: 38 · 2016: 8
- FWCI
- 170.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- SAGE Open (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Keywords
- Mechanical Turk, sample representativeness, demographic structure, political composition, partisanship, American National Election Studies, replication, survey sampling bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Demographic and Political Composition of Mechanical Turk Samples. https://app.cochat.ai/papers/10.1177/2158244016636433
- The paper
- Levay, K. E., Freese, J., & Druckman, J. N. (2016). The Demographic and Political Composition of Mechanical Turk Samples. Sage Open. https://doi.org/10.1177/2158244016636433
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.