“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
As Amazon’s Mechanical Turk (MTurk) has surged in popularity throughout political science, scholars have increasingly challenged the external validity of inferences made drawing upon MTurk samples. At workshops and conferences experimental and survey-based researchers hear questions about the demographic characteristics, political preferences, occupation, and geographic location of MTurk respondents. In this paper we answer these questions and present a number of novel results. By introducing a new benchmark comparison for MTurk surveys, the Cooperative Congressional Election Survey, we compare the joint distributions of age, gender, and race among MTurk respondents within the United States. In addition, we compare political, occupational, and geographical information about respondents from MTurk and CCES. Throughout the paper we show several ways that political scientists can use the strengths of MTurk to attract respondents with specific characteristics of interest to best answer their substantive research questions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, we must recognize potential drawbacks to using MTurk (Huff and Tingley, 2015).”
From Virtual reality adoption during the COVID-19 pandemic: A uses and gratifications perspective (2021) · cites this paper as Huff and Tingley, 2015
In the citing paper
“Moreover, MTurk samples are found to be representative for the US population and strong in external validity (Huff & Tingley, 2015).”
From Consumers Beware: Online Personalized Pricing in Action! How the Framing of a Mandated Discriminatory Pricing Disclosure Influences Intention to Purchase (2020) · cites this paper as Huff & Tingley, 2015
In the citing paper
“Compared to the U.S. population, our sample is younger and more educated, a common characteristic of samples recruited through online crowdsourcing platforms [43, 45, 81, 88].”
From "I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust (2024) · cites this paper as [45]
Citations
Cites
The Nature and Origins of Mass Opinion
Zaller, J. R. (1992) · doi:10.1017/cbo9780511818691 · Cited by 7,785
The Nature and Origins of Mass Opinion.
Kenski, H. C., & Zaller, J. R. (1993) · Contemporary Sociology · doi:10.2307/2074664 · Cited by 5,169
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · cited 16×
“The comparisons presented by Berinsky et al. (2012) provide an excellent foundation for exploring the relationship between samples drawn from MTurk surveys and other subject pools commonly used by political scientists.1 In this paper we present a number of results that contribute to a broader goal of understanding sur…”
BackgroundMethodInfluentialInside the Turk: Understanding Mechanical Turk as a Participant Pool
Paolacci, G., & Chandler, J. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414531598 · Cited by 2,136
Peterson, R. A. (2001) · Journal of Consumer Research · doi:10.1086/323732 · Cited by 1,258 · cited 1×
“For example, scholars have frequently debated the quality of inferences when the results are drawn from college-age convenience samples (Druckman and Kam, 2011; Peterson, 2001).”
BackgroundWho are the crowdworkers?: shifting demographics in mechanical turk
Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010) · CHI '10 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/1753846.1753873 · Cited by 1,004 · cited 2×
“(2004), Zaller (1992), and Bamundo and Kopelman (1980).”
BackgroundNews vs. Entertainment: How Increasing Media Choice Widens Gaps in Political Knowledge and Turnout
Prior, M. (2005) · American Journal of Political Science · doi:10.1111/j.1540-5907.2005.00143.x · Cited by 955 · cited 1×
“This means that MTurk could be used by political scientists to build upon prior research exploring the complex relationship between news interest, political knowledge, and voter turnout (Philpot, 2004; Prior, 2005; Zaller, 1992).”
MethodBerinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 1×
“Berinsky et al. (2013) shows that as many as half of all respondents breeze through surveys without paying attention and that Screener passage correlates with politically relevant characteristics, such as education and race.”
BackgroundChandler, J., Mueller, P., & Paolacci, G. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0365-7 · Cited by 950 · cited 4×
“In the last several years Amazon’s Mechanical Turk (MTurk) has surged in popularity in experimental and survey-based social science research (Berinsky et al., 2012; Chandler et al., 2014; Krupnikov and Levine, 2014; Paolacci and Chandler, 2014).”
BackgroundInfluentialCambridge Handbook of Experimental Political Science
(2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511921452 · Cited by 611
Students as Experimental Participants: A Defense of the “Narrow Data Base”
Druckman, J. N., & Kam, C. D. (2011) · Cambridge Handbook of Experimental Political Science · doi:10.1017/cbo9780511921452.004 · Cited by 454 · cited 1×
“For example, scholars have frequently debated the quality of inferences when the results are drawn from college-age convenience samples (Druckman and Kam, 2011; Peterson, 2001).”
BackgroundAmerican political science review
(1983) · Electoral Studies · doi:10.1016/0261-3794(83)90125-7 · Cited by 445
Fischer, A. R., & Shaw, C. M. (1999) · Journal of Counseling Psychology · doi:10.1037/0022-0167.46.3.395 · Cited by 437
Making Sense of Issues Through Media Frames: Understanding the Kosovo Crisis
Berinsky, A. J., & Kinder, D. R. (2006) · The Journal of Politics · doi:10.1111/j.1468-2508.2006.00451.x · Cited by 332 · cited 2×
“(2012) reach this conclusion by comparing MTurk to convenience samples from prior work (Berinsky and Kinder, 2006; Kam et al., 2007) and the American National Election 2008– 2009 Panel Study.”
MethodResultDepressive mood and obesity in US adults: comparison and moderation by sex, age, and race
Heo, M., Pietrobelli, A., Fontaine, K. R., Sirey, J. A., & Faith, M. S. (2006) · International Journal of Obesity · doi:10.1038/sj.ijo.0803122 · Cited by 330 · cited 1×
“For examples of the potential moderating effects of race, age, gender, political party, and occupation see Fischer and Shaw (1999), Heo et al. (2005) We obtain similar results if we analyze the collection of team surveys which has a sample size of over 50,000 respondents.”
ResultGRIMMER, J., MESSING, S., & WESTWOOD, S. J. (2012) · American Political Science Review · doi:10.1017/s0003055412000457 · Cited by 296 · cited 2×
“Researchers have used the results from MTurk surveys to answer a wide array of questions ranging from understanding the limitations of voters to exploring cognitive biases and the strengths of political arguments (Arceneaux, 2012; Grimmer et al., 2012; Huber et al., 2012).”
HUBER, G. A., HILL, S. J., & LENZ, G. S. (2012) · American Political Science Review · doi:10.1017/s0003055412000391 · Cited by 252 · cited 1×
“Researchers have used the results from MTurk surveys to answer a wide array of questions ranging from understanding the limitations of voters to exploring cognitive biases and the strengths of political arguments (Arceneaux, 2012; Grimmer et al., 2012; Huber et al., 2012).”
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251 · cited 5×
“27 A recent strain of research exploring the characteristics of MTurk workers argues that what differentiates MTurkers is their status as permanent participants (Chandler et al., 2014; Krupnikov and Levine, 2014; Paolacci and Chandler, 2014).”
BackgroundResultInfluentialCognitive Biases and the Strength of Political Arguments
Arceneaux, K. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00573.x · Cited by 244 · cited 4×
“Researchers have used the results from MTurk surveys to answer a wide array of questions ranging from understanding the limitations of voters to exploring cognitive biases and the strengths of political arguments (Arceneaux, 2012; Grimmer et al., 2012; Huber et al., 2012). As this type of work has grown in popularity,…”
BackgroundResultInfluentialLABOR MARKET MOBILITY AND CASH COMPENSATION: THE MODERATING EFFECTS OF RACE AND GENDER.
Dreher, G. F., & Cox, T. H. (2000) · Academy of Management Journal · doi:10.2307/1556417 · Cited by 167
Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research
Kam, C. D., Wilking, J. R., & Zechmeister, E. J. (2007) · Political Behavior · doi:10.1007/s11109-007-9037-6 · Cited by 158 · cited 2×
“(2012) reach this conclusion by comparing MTurk to convenience samples from prior work (Berinsky and Kinder, 2006; Kam et al., 2007) and the American National Election 2008– 2009 Panel Study.”
MethodResultBamundo, P. J., & Kopelman, R. E. (1980) · Journal of Vocational Behavior · doi:10.1016/0001-8791(80)90020-2 · Cited by 88 · cited 1×
“(2004), Zaller (1992), and Bamundo and Kopelman (1980).”
A Party of a Different Color? Race, Campaign Communication, and Party Politics
Philpot, T. S. (2004) · Political Behavior · doi:10.1023/b:pobe.0000043455.25490.a9 · Cited by 26 · cited 1×
“This means that MTurk could be used by political scientists to build upon prior research exploring the complex relationship between news interest, political knowledge, and voter turnout (Philpot, 2004; Prior, 2005; Zaller, 1992).”
MethodThe Usurped Powers of the Senate
Low, A. M. (1906) · American Political Science Review · doi:10.2307/1944843 · Cited by 18
Research works printed but not linked: 3
- Gay C, 2015, Journal of Race, Ethnicity, and Politics
- Leeper T, 2013, The Political Methodologist, 20, 2
- Leeper TJ, 2015, 6, 5, 1
Cited by
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,228
Conducting Clinical Research Using Crowdsourced Convenience Samples
Chandler, J., & Shapiro, D. (2016) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-021815-093623 · Cited by 1,072
“Interestingly, measures of household income are closer to national averages (Greenblatt 2013, Huff & Tingley 2014) and workers are less likely to live alone (Greenblatt 2013, 2014).”
BackgroundInfluentialKees, J., Berry, C., Burton, S., & Sheehan, K. (2017) · Journal of Advertising · doi:10.1080/00913367.2016.1269304 · Cited by 902
“Some large-scale, survey-based studies have found that MTurk respondents are not too different from respondents on other survey platforms (Huff and Tingley 2015), but others have identified some differences in age, education, gender, and income (Maynard 2014; Peer et al. 2015).”
BackgroundAmericans’ COVID-19 stress, coping, and adherence to CDC guidelines
Park, C. L., Russell, B. S., Fendrich, M., Finkelstein-Fox, L., Hutchison, M., & Becker, J. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-05898-9 · Cited by 728
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R. M., & Waggoner, P. D. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 686
“Researchers consistently find that MTurk samples tend to be more politically liberal, younger, less religious, and less racially diverse than the U.S. population (Berinsky et al. 2012; Huff and Tingley, 2015).”
Background
Metadata
- Authors
- Huff, C., & Tingley, D.
- Affiliations
- Harvard University (US)
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2015-07-01
- Volume
- 2
- Issue
- 3
- ISSN
- 2053-1680 (ISSN-L)
- OpenAlex
- W1936420698
- Cited by
- 903
- References
- 25
- Citations by year
- 2026: 25 · 2025: 41 · 2024: 53 · 2023: 71 · 2022: 111 · 2021: 156 · 2020: 112 · 2019: 106 · 2018: 107 · 2017: 81 · 2016: 34 · 2015: 2
- FWCI
- 112.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Research & Politics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Quantitative Social Science, Harvard University
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Political Science and International Relations › Electoral Systems and Political Participation
- Keywords
- Mechanical Turk, generalizability, respondent demographics, political preferences, job characteristics, geographic location
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.