Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics
Open accessAbstract
This article examines online recruitment via Facebook, Mechanical Turk (MTurk), and Qualtrics panels in India and the United States. It compares over 7300 respondents—1000 or more from each source and country—to nationally representative benchmarks in terms of demographics, political attitudes and knowledge, cooperation, and experimental replication. In the United States, MTurk offers the cheapest and fastest recruitment, Qualtrics is most demographically and politically representative, and Facebook facilitates targeted sampling. The India samples look much less like the population, though Facebook offers broad geographical coverage. We find online convenience samples often provide valid inferences into how partisanship moderates treatment effects. Yet they are typically unrepresentative on such political variables, which has implications for the external validity of sample average treatment effects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MTurk is not representative, particularly when it comes to self-reported political identity and ideology.
“and is, in fact, not representative, particularly when it comes to self-reported political identity and ideology51–53”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [52]
MTurk offers convenience samples and makes no attempt to be nationally representative.
“MTurk offers convenience samples and makes no attempt to be nationally representative”
From Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008 · see citing works
Running experiments on Amazon Mechanical Turk
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010) · Judgment and Decision Making · doi:10.1017/s1930297500002205 · Cited by 3,767 · see citing works
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618 · see citing works
The Generalizability of Survey Experiments
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Jeremy Freese, A. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2015.19 · Cited by 1,235 · see citing works
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014) · American Journal of Political Science · doi:10.1111/ajps.12081 · Cited by 951 · cited 6× · see citing works
“…rates for MTurk workers ranged from 85% to 97% (Peer, Vosgerau and Acquisti, 2014; Litman, Robinson and Rosenzweig, 2014), higher than the 65% obtained by Berinsky, Margolis and Sances (2014) Figure 7: India: News Consumption and Political Knowledge l l l l l l l Unweighted 0 0.25 0.5 0.75 1”
BackgroundMethodResultInfluentialHuff, C., & Tingley, D. (2015) · Research & Politics · doi:10.1177/2053168015604648 · Cited by 903 · see citing works
Are samples drawn from Mechanical Turk valid for research on political ideology?
Clifford, S., Jewell, R., & Waggoner, P. (2015) · Research & Politics · doi:10.1177/2053168015622072 · Cited by 680 · see citing works
The demographic and political composition of Mechanical Turk samples
Levay, K. E., Freese, J., & Druckman, J. N. (2016) · Sage Open · doi:10.1177/2158244016636433 · Cited by 676 · see citing works
Grimmer, J., Messing, S., & Westwood, S. J. (2012) · American Political Science Review · doi:10.1017/s0003055412000457 · Cited by 296 · see citing works
Litman, L., Robinson, J., & Rosenzweig, C. (2014) · Behavior Research Methods · doi:10.3758/s13428-014-0483-x · Cited by 275 · see citing works
The Returns to English-Language Skills in India
Azam, M., Chin, A. Y., & Prakash, N. (2012) · Economic Development and Cultural Change · doi:10.1086/668277 · Cited by 274 · see citing works
The Power of Partisanship in Brazil: Evidence from Survey Experiments
Samuels, D., & Zucco, C. (2013) · American Journal of Political Science · doi:10.1111/ajps.12050 · Cited by 259 · see citing works
Huber, G. A., Hill, S. J., & Lenz, G. (2012) · American Political Science Review · doi:10.1017/s0003055412000391 · Cited by 253 · see citing works
Cross-Sample Comparisons and External Validity
Krupnikov, Y., & Levine, A. S. (2014) · Journal of Experimental Political Science · doi:10.1017/xps.2014.7 · Cited by 251 · see citing works
Cognitive Biases and the Strength of Political Arguments
Arceneaux, K. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00573.x · Cited by 244 · see citing works
Christenson, D. P., & Glick, D. M. (2014) · American Journal of Political Science · doi:10.1111/ajps.12150 · Cited by 226 · see citing works
Self-citationAmericans' Belief in Linked Fate: Does the Measure Capture the Concept?
Gay, C., Hochschild, J. L., & White, A. R. (2016) · The Journal of Race Ethnicity and Politics · doi:10.1017/rep.2015.3 · Cited by 209 · see citing works
Ding, Y. (2009) · 北京周报：英文版 · Cited by 206 · see citing works
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0157732 · Cited by 161 · see citing works
How naming and shaming affects human rights perceptions in the shamed country
Ausderan, J. (2014) · Journal of Peace Research · doi:10.1177/0022343313510014 · Cited by 158 · see citing works
Charnysh, V., Lucas, C., & Singh, P. (2014) · Comparative Political Studies · doi:10.1177/0010414014543103 · Cited by 131 · see citing works
Social desirability bias and self-reports of motivation
Antin, J., & Shaw, A. (2012) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208699 · Cited by 125 · see citing works
Comparisons of Online Recruitment Strategies for Convenience Samples
Antoun, C., Zhang, C., Conrad, F. G., & Schober, M. F. (2015) · Field Methods · doi:10.1177/1525822x15603149 · Cited by 124 · see citing works
Foreign Aid and Government Legitimacy
Dietrich, S., & Winters, M. S. (2015) · Journal of Experimental Political Science · doi:10.1017/xps.2014.31 · Cited by 100 · see citing works
Using Facebook as a Subject Recruitment Tool for Survey-Experimental ResearchPreprint
Samuels, D., & Zucco, C. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2101458 · Cited by 78 · see citing works
Public Opinion, Policy Tools, and the Status Quo
Haselswerdt, J., & Bartels, B. L. (2015) · Political Research Quarterly · doi:10.1177/1065912915591217 · Cited by 78 · see citing works
Christenson, D. P., & Glick, D. M. (2015) · Public Opinion Quarterly · doi:10.1093/poq/nfv040 · Cited by 67 · see citing works
Self-citationConscription, Inequality, and Partisan Support for War
Kriner, D. L., & Shen, F. X. (2015) · Journal of Conflict Resolution · doi:10.1177/0022002715590877 · Cited by 47 · see citing works
Pastors for Pinochet: Authoritarian Stereotypes and Voting for Evangelicals in Chile
Boas, T. C. (2016) · Journal of Experimental Political Science · doi:10.1017/xps.2015.17 · Cited by 47 · see citing works
Self-citationSantoso, L. P., Stein, R. M., & Stevenson, R. T. (2016) · Political Analysis · doi:10.1093/pan/mpw016 · Cited by 25 · see citing works
The double-edged sword of mobilizing citizens via mobile phone in developing countries
Erlich, A., Jung, D. F., Long, J. D., & McIntosh, C. (2017) · Development Engineering · doi:10.1016/j.deveng.2017.11.001 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Christenson DP and Glick DM (2013) Crowdsourcing Panel Studies and Real-Time Experiments in MTurk. The Political Methodologist 20(2), 27-33.
Cited by
Sorted by citations
Pennycook, G., & Rand, D. G. (2022) · Nature Communications · doi:10.1038/s41467-022-30073-5 · Cited by 270 · see citing works
Nelson, B. W., Pettitt, A., Flannery, J. E., & Allen, N. B. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241990 · Cited by 175 · see citing works
“First, online recruitment has been found to have variable demographic and political representation [41].”
BackgroundSmith, M. E., Sharpe, T. L., Richardson, J., Pahwa, R., Smith, D., & DeVylder, J. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.112587 · Cited by 147 · see citing works
Arceneaux, K., & Truex, R. (2023) · Perspectives on Politics · doi:10.1017/s1537592722000901 · Cited by 97 · see citing works
“Importantly, the data quality and representativeness of Qualtrics samples have been independently verified by scholars to be in line with probability samples like the General Social Survey and the American National Election Studies (Boas, Christenson and Glick, 2020).”
Davis, M., Gilbar, O., & Padilla-Medina, D. (2020) · doi:10.1101/2020.06.08.20125914 · Cited by 61 · see citing works
COVID-19 frauds: an exploratory study of victimization during a global crisis
Kennedy, J. P., Rorie, M., & Benson, M. L. (2021) · Criminology & Public Policy · doi:10.1111/1745-9133.12554 · Cited by 59 · see citing works
Lewin, K. M., Meshi, D., Schuster, A. M., & Cotten, S. R. (2023) · Aging & Mental Health · doi:10.1080/13607863.2022.2068133 · Cited by 37 · see citing works
Rogge, R. D., Lin, Y.-Y., Swanson, D. P., & Amaro, A. (2024) · Mindfulness · doi:10.1007/s12671-022-01886-8 · Cited by 25 · see citing works
A daily diary study into the effects on mental health of COVID-19 pandemic-related behaviors
Shaw, P., Blizzard, S., Shastri, G., Kundzicz, P., Curtis, B., Ungar, L., & Koehly, L. (2021) · Psychological Medicine · doi:10.1017/s0033291721001896 · Cited by 14 · see citing works
Mindfulness and Depersonalization: a Nuanced Relationship
Levin, K. K., Gornish, A., & Quigley, L. (2022) · Mindfulness · doi:10.1007/s12671-022-01890-y · Cited by 15 · see citing works
“Moreover, studies support the representativeness and quality of data collected through Qualtrics over other online platforms (Boas et al., 2020; Heen et al., 2014; Ibarra et al., 2018).”
BackgroundWhy do cognitive prompts hurt learning in older adults?
Kraiger, K., Cavanagh, T. M., & Willis, C. M. G. (2020) · International Journal of Training and Development · doi:10.1111/ijtd.12169 · Cited by 11 · see citing works
Milkovich, L. M., Randell, K. A., Sherman, A. K., & Moreno, M. A. (2025) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0000000000001328 · Cited by 10 · see citing works
Respect Your Elders: Generativity and Life Satisfaction in Caregiving Grandparents
Scott, R. K., Nadorff, D. K., Barnett, M., & Yancura, L. (2023) · The International Journal of Aging and Human Development · doi:10.1177/00914150221092128 · Cited by 10 · see citing works
Shyrokonis, Y., Peitzmeier, S., Ward, M., Fedina, L., Tolman, R., & Herrenkohl, T. I. (2025) · Violence Against Women · doi:10.1177/10778012231222491 · Cited by 4 · see citing works
- Authors
- Boas, T. C., Christenson, D. P., & Glick, D. M.
- Affiliations
- Boston University (US)
- Venue
- Political Science Research and Methods
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-08-08
- Volume
- 8
- Issue
- 2
- Pages
- 232-250
- ISSN
- 2049-8470 (ISSN-L)
- 2049-8489
- OpenAlex
- W2886505161
- Cited by
- 570
- References
- 32
- Citations by year
- 2026: 25 · 2025: 94 · 2024: 77 · 2023: 103 · 2022: 108 · 2021: 99 · 2020: 48 · 2019: 15 · 2018: 1
- FWCI
- 93.66
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Political Science Research and Methods (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- OpenBU (Boston University) (submittedVersion)
- Funders
- University of Cambridge
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Communication › Social Media and Politics
- Keywords
- online recruitment, Facebook, Mechanical Turk, convenience samples, sample representativeness, political attitudes, political knowledge, cooperation, experimental replication, generalizability, partisanship, United States, India
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
