The Demographic and Political Composition of Mechanical Turk Samples
Open accessThe DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.
Abstract
One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Market research quota sampling is a non-probabilistic sampling strategy whereby participants are identified from a database to match the population of interest (a discussion of the use of these estimates is in the appendix [p 7]).10 Samples obtained via this method might produce results that differ from the general population.11, 12”
From Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic (2021) · cites this paper as [12]
In the citing paper
“MTurk -based samples have also been criticized as being younger, more educated, and less politically and religiously diverse than the general population (Levay et al., 2016), potentially limiting generalizability.”
From Cultural humility and cultural competence in counseling: An exploratory mixed methods investigation (2023) · cites this paper as Levay et al., 2016
Citations
Cites
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011) · Perspectives on Psychological Science · doi:10.1177/1745691610393980 · Cited by 9,996
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · cited 20×
“Following others (e.g., Berinsky et al., 2012; Huff & Tingley, 2015; Krupnikov & Levine, 2014), we set three requirements for participation: a HIT Approval Rate greater than or equal to 90, at least 50 HITs approved, and an IP address based in the United States.”
BackgroundMethodResultInfluentialRunning experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · cited 3×
“…with 45% of the MTurk sample possessing at least a bachelor’s degree versus only 29.5% of the ANES sample.8 Interestingly, while some earlier work highlights that MTurk samples underrepresent men (e.g., Ipeirotis, 2010; Paolacci et al., 2010), our data exhibit the opposite pattern (53.9% male).”
BackgroundMethodResultInfluentialInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Couper, M. P. (2000) · Public Opinion Quarterly · doi:10.1086/318641 · Cited by 2,078
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · cited 2×
“This finding, however, is consistent with more recent work (e.g., Berinsky et al., 2012; Casler et al., 2013; Krupnikov & Levine, 2014).”
MethodResultThe Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228 · cited 6×
“…& Levine, 2014; Paolacci & Chandler, 2014).3 Similarly, differences in political identity appear to be mostly stable, with MTurk samples frequently containing many more Democrats and liberals than populationbased samples (Berinsky et al., 2012; Krupnikov & Levine, 2014; Mullinix et al., 2015).”
BackgroundMethodInfluentialSelf-citationHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · cited 11×
“With the above in mind, as highlighted by others (e.g., Huff & Tingley, 2015), a promising approach for researchers concerned with representativeness or with specific sample needs is to employ software and sampling procedures aimed at creating larger and/or targeted survey pools.”
BackgroundMethodResultInfluentialThe Unheavenly Chorus: Unequal Political Voice and the Broken Promise of American Democracy
Schlozman, K. L., Verba, S., & Brady, H. E. (2012) · Princeton University Press eBooks · doi:10.1515/9781400841912 · Cited by 701
Population-Based Survey Experiments
Mutz, D. C. (2011) · doi:10.1515/9781400840489 · Cited by 674
Research synthesis: AAPOR report on online panels
Prepared for the AAPOR Executive Council by a Task Force operating under the auspices of the AAPOR Standards Committee, W. M. I., Baker, R., Blumberg, S. J., Brick, J. M., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D., Frankel, M. R., Garland, P., et al. (2010) · Public Opinion Quarterly · doi:10.1093/poq/nfq048 · Cited by 584 · cited 4×
“This also means calculating respondents’ chances of being sampled is virtually impossible (see Baker et al., 2010; Couper, 2011).”
BackgroundInfluentialFinancial incentives and the "performance of crowds"
Mason, W., & Watts, D. J. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600175 · Cited by 520
Demographics of Mechanical Turk
Ipeirotis, P. G. (2010) · Faculty Digital Archive (New York University Florence) · Cited by 494
The Future of Modes of Data Collection
Couper, M. P. (2011) · Public Opinion Quarterly · doi:10.1093/poq/nfr046 · Cited by 309 · cited 2×
“This also means calculating respondents’ chances of being sampled is virtually impossible (see Baker et al., 2010; Couper, 2011).”
BackgroundHUBER, G. A., HILL, S. J., & LENZ, G. S. (2012) · American Political Science Review · doi:10.1017/s0003055412000391 · Cited by 252
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251 · cited 5×
“…ages, and much smaller percentages of most nonWhite groups, especially Blacks (Berinsky et al., 2012; Buhrmester et al., 2011; Huff & Tingley, 2015; Ipeirotis, 2010; Krupnikov & Levine, 2014; Paolacci & Chandler, 2014).3 Similarly, differences in political identity appear to be mostly stable, with…”
BackgroundResultInfluentialCognitive Biases and the Strength of Political Arguments
Arceneaux, K. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00573.x · Cited by 244 · cited 1×
“…allows researchers to easily recruit participants and administer surveys at little cost: Consequently, it has emerged as a prominent data source among researchers of U.S. public opinion (e.g., Arceneaux, 2012; Gerber, Huber, Doherty, & Dowling, 2011; Huber, Hill, & Lenz, 2012; Huber & Paris, 2013).”
BackgroundOnline Panel Research: Data Quality Perspective, A
(2014) · doi:10.1002/9781118763520 · Cited by 184
Bohannon, J. (2011) · Science · doi:10.1126/science.334.6054.307 · Cited by 140
Online panel research: History, concepts, applications and a look at the future
Callegaro, M., Baker, R., Bethlehem, J. G., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (2014) · Cited by 71
Selectively Recruiting Participants from Amazon Mechanical Turk Using QualtricsPreprint
Péer, E., Paolacci, G., Chandler, J., & Mueller, P. A. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2100631 · Cited by 67
Assessing the Programmatic Equivalence Assumption in Question Wording Experiments
Huber, G. A., & Paris, C. (2013) · Public Opinion Quarterly · doi:10.1093/poq/nfs054 · Cited by 65 · cited 1×
“…allows researchers to easily recruit participants and administer surveys at little cost: Consequently, it has emerged as a prominent data source among researchers of U.S. public opinion (e.g., Arceneaux, 2012; Gerber, Huber, Doherty, & Dowling, 2011; Huber, Hill, & Lenz, 2012; Huber & Paris, 2013).”
BackgroundLiu, M., & Wang, Y. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.01.057 · Cited by 63 · cited 1×
“…completion of the survey via Internet by an independently drawn, population-based sample, we can analyze and compare the data from the 2012 Web ANES sample with data from an MTurk sample while controlling for survey mode (see Struminskaya et al., 2014 and Liu & Wang, 2015 on survey mode effects).”
MethodGerber, A. S., Huber, G. A., Doherty, D., & Dowling, C. M. (2011) · The Journal of Politics · doi:10.1017/s0022381611000892 · Cited by 27
Assessing representativeness of a probability‐based online panel in Germany
Struminskaya, B., Kaczmirek, L., Schaurer, I., & Bandilla, W. (2014) · doi:10.1002/9781118763520.ch3 · Cited by 26
Research works printed but not linked: 2
- Leeper T. J., 2013, The Political Methodologist, 2, 2
- Mason W. A., 2009, Association for Computing Machinery Explorations Newsletter, 11, 100
Cited by
Santomauro, D. F., Mantilla Herrera, A. M., Shadid, J., Zheng, P., Ashbaugh, C., Pigott, D. M., Abbafati, C., Adolph, C., Amlag, J. O., Aravkin, A. Y., et al. (2021) · The Lancet · doi:10.1016/s0140-6736(21)02143-7 · Cited by 5,444
Coppock, A., & McClellan, O. A. (2019) · Research & Politics · doi:10.1177/2053168018822174 · Cited by 987
“…work, we need probability samples, but for research that models the relationships between variables, convenience samples may be acceptable.2 Levay et al. (2016) provides some empirical support for the compromise’s underlying reasoning: MTurk and probability samples are descriptively quite…”
BackgroundMTurk Research: Review and Recommendations
Aguinis, H., Villamor, I., & Ramani, R. S. (2021) · Journal of Management · doi:10.1177/0149206320969787 · Cited by 984
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01273-7 · Cited by 668
Crowdsourcing Consumer Research
Goodman, J. K., & Paolacci, G. (2017) · Journal of Consumer Research · doi:10.1093/jcr/ucx047 · Cited by 588
“…model-based poststratification can be used to statistically adjust the estimates obtained from nonrepresentative samples (Battaglia, Hoaglin, and Frankel 2013; Park, Gelman, and Bafumi 2004), and may be applied to MTurk samples (Goel, Obeng, and Rothschild 2015; Levay, Freese, and Druckman 2016).”
Background
Metadata
- Authors
- Levay, K. E., Freese, J., & Druckman, J. N.
- Affiliations
- Northwestern University (US)
- Stanford University, CA, USA
- Corresponding author
- Druckman, James N.
- ORCID
- Druckman, James N.
- Venue
- Sage Open
- Publisher
- SAGE Publications
- Published
- 2016-01-01
- Volume
- 6
- Issue
- 1
- ISSN
- 2158-2440 (ISSN-L)
- OpenAlex
- W2301926123
- Cited by
- 676
- References
- 26
- Citations by year
- 2026: 21 · 2025: 48 · 2024: 58 · 2023: 75 · 2022: 88 · 2021: 103 · 2020: 103 · 2019: 72 · 2018: 61 · 2017: 38 · 2016: 8
- FWCI
- 170.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- SAGE Open (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Keywords
- Mechanical Turk, sample representativeness, demographic structure, political composition, partisanship, American National Election Studies, replication, survey sampling bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.