“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
As Amazon’s Mechanical Turk (MTurk) has surged in popularity throughout political science, scholars have increasingly challenged the external validity of inferences made drawing upon MTurk samples. At workshops and conferences experimental and survey-based researchers hear questions about the demographic characteristics, political preferences, occupation, and geographic location of MTurk respondents. In this paper we answer these questions and present a number of novel results. By introducing a new benchmark comparison for MTurk surveys, the Cooperative Congressional Election Survey, we compare the joint distributions of age, gender, and race among MTurk respondents within the United States. In addition, we compare political, occupational, and geographical information about respondents from MTurk and CCES. Throughout the paper we show several ways that political scientists can use the strengths of MTurk to attract respondents with specific characteristics of interest to best answer their substantive research questions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, we must recognize potential drawbacks to using MTurk (Huff and Tingley, 2015).”
From Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective (2021) · cites this paper as Huff and Tingley, 2015
In the citing paper
“Moreover, MTurk samples are found to be representative for the US population and strong in external validity (Huff & Tingley, 2015).”
From Consumers Beware: Online Personalized Pricing in Action! How the Framing of a Mandated Discriminatory Pricing Disclosure Influences Intention to Purchase (2020) · cites this paper as Huff & Tingley, 2015
In the citing paper
“Compared to the U.S. population, our sample is younger and more educated, a common characteristic of samples recruited through online crowdsourcing platforms [43, 45, 81, 88].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [45]
Citations
Cites
The Nature and Origins of Mass Opinion
Zaller, J. R. (1992) · doi:10.1017/cbo9780511818691 · Cited by 7,785
The Nature and Origins of Mass Opinion.
Kenski, H. C., & Zaller, J. R. (1993) · Contemporary Sociology · doi:10.2307/2074664 · Cited by 5,169
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · cited 16×
“The comparisons presented by Berinsky et al. (2012) provide an excellent foundation for exploring the relationship between samples drawn from MTurk surveys and other subject pools commonly used by political scientists.1 In this paper we present a number of results that contribute to a broader goal of understanding sur…”
BackgroundMethodInfluentialInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · cited 1×
“For example, scholars have frequently debated the quality of inferences when the results are drawn from college-age convenience samples (Druckman and Kam, 2011; Peterson, 2001).”
BackgroundWho are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004 · cited 2×
“(2004), Zaller (1992), and Bamundo and Kopelman (1980).”
BackgroundNews vs. Entertainment: How Increasing Media Choice Widens Gaps in Political Knowledge and Turnout
Prior, M. (2005) · American Journal of Political Science · doi:10.1111/j.1540-5907.2005.00143.x · Cited by 955 · cited 1×
“This means that MTurk could be used by political scientists to build upon prior research exploring the complex relationship between news interest, political knowledge, and voter turnout (Philpot, 2004; Prior, 2005; Zaller, 1992).”
MethodBerinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“Berinsky et al. (2013) shows that as many as half of all respondents breeze through surveys without paying attention and that Screener passage correlates with politically relevant characteristics, such as education and race.”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · cited 4×
“In the last several years Amazon’s Mechanical Turk (MTurk) has surged in popularity in experimental and survey-based social science research (Berinsky et al., 2012; Chandler et al., 2014; Krupnikov and Levine, 2014; Paolacci and Chandler, 2014).”
BackgroundInfluentialCambridge Handbook of Experimental Political Science
(2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452 · Cited by 611
Research works printed but not linked: 3
- Gay C, 2015, Journal of Race, Ethnicity, and Politics
- Leeper T, 2013, The Political Methodologist, 20, 2
- Leeper TJ, 2015, 6, 5, 1
Cited by
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
“Interestingly, measures of household income are closer to national averages (Greenblatt 2013, Huff & Tingley 2014) and workers are less likely to live alone (Greenblatt 2013, 2014).”
BackgroundInfluentialKees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
“Some large-scale, survey-based studies have found that MTurk respondents are not too different from respondents on other survey platforms (Huff and Tingley 2015), but others have identified some differences in age, education, gender, and income (Maynard 2014; Peer et al. 2015).”
BackgroundAmericans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 686
“Researchers consistently find that MTurk samples tend to be more politically liberal, younger, less religious, and less racially diverse than the U.S. population (Berinsky et al. 2012; Huff and Tingley, 2015).”
Background
Metadata
- Authors
- Huff, C., & Tingley, D.
- Affiliations
- Harvard University (US)
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2015-07-01
- Volume
- 2
- Issue
- 3
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W1936420698
- Cited by
- 903
- References
- 25
- Citations by year
- 2026: 25 · 2025: 41 · 2024: 53 · 2023: 71 · 2022: 111 · 2021: 156 · 2020: 112 · 2019: 106 · 2018: 107 · 2017: 81 · 2016: 34 · 2015: 2
- FWCI
- 112.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Research & Politics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Quantitative Social Science, Harvard University
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Political Science and International Relations › Electoral Systems and Political Participation
- Keywords
- Mechanical Turk, generalizability, respondent demographics, political preferences, job characteristics, geographic location
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. https://app.cochat.ai/papers/10.1177/2053168015604648
- The paper
- Huff, C., & Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics. https://doi.org/10.1177/2053168015604648
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.